OTBN Simulation Results

Wednesday July 03 2024 23:02:32 UTC

GitHub Revision: `e6706fcc7b`

Branch: os_regression

Simulator: XCELIUM

Build randomization enabled with --build-seed 8083624550445280117614176890238357255195852125596561370221115831648066795492

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	otbn_smoke	13.000s	67.988us	1	1	100.00
V1	single_binary	otbn_single	48.000s	192.512us	100	100	100.00
V1	csr_hw_reset	otbn_csr_hw_reset	6.000s	18.587us	5	5	100.00
V1	csr_rw	otbn_csr_rw	5.000s	15.312us	20	20	100.00
V1	csr_bit_bash	otbn_csr_bit_bash	8.000s	230.472us	5	5	100.00
V1	csr_aliasing	otbn_csr_aliasing	7.000s	26.836us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	otbn_csr_mem_rw_with_rand_reset	9.000s	67.778us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	otbn_csr_rw	5.000s	15.312us	20	20	100.00
		otbn_csr_aliasing	7.000s	26.836us	5	5	100.00
V1	mem_walk	otbn_mem_walk	53.000s	1.855ms	5	5	100.00
V1	mem_partial_access	otbn_mem_partial_access	26.000s	364.319us	5	5	100.00
V1		TOTAL			166	166	100.00
V2	reset_recovery	otbn_reset	40.000s	139.334us	10	10	100.00
V2	multi_error	otbn_multi_err	54.000s	300.154us	1	1	100.00
V2	back_to_back	otbn_multi	1.867m	1.155ms	10	10	100.00
V2	stress_all	otbn_stress_all	2.417m	383.179us	10	10	100.00
V2	lc_escalation	otbn_escalate	24.000s	48.341us	52	60	86.67
V2	zero_state_err_urnd	otbn_zero_state_err_urnd	9.000s	275.377us	4	5	80.00
V2	sw_errs_fatal_chk	otbn_sw_errs_fatal_chk	25.000s	87.210us	10	10	100.00
V2	alert_test	otbn_alert_test	6.000s	52.233us	50	50	100.00
V2	intr_test	otbn_intr_test	9.000s	39.054us	50	50	100.00
V2	tl_d_oob_addr_access	otbn_tl_errors	10.000s	186.623us	20	20	100.00
V2	tl_d_illegal_access	otbn_tl_errors	10.000s	186.623us	20	20	100.00
V2	tl_d_outstanding_access	otbn_csr_hw_reset	6.000s	18.587us	5	5	100.00
		otbn_csr_rw	5.000s	15.312us	20	20	100.00
		otbn_csr_aliasing	7.000s	26.836us	5	5	100.00
		otbn_same_csr_outstanding	7.000s	369.556us	20	20	100.00
V2	tl_d_partial_access	otbn_csr_hw_reset	6.000s	18.587us	5	5	100.00
		otbn_csr_rw	5.000s	15.312us	20	20	100.00
		otbn_csr_aliasing	7.000s	26.836us	5	5	100.00
		otbn_same_csr_outstanding	7.000s	369.556us	20	20	100.00
V2		TOTAL			237	246	96.34
V2S	mem_integrity	otbn_imem_err	16.000s	31.601us	10	10	100.00
		otbn_dmem_err	15.000s	35.729us	15	15	100.00
V2S	internal_integrity	otbn_alu_bignum_mod_err	14.000s	122.070us	5	5	100.00
		otbn_controller_ispr_rdata_err	12.000s	118.996us	5	5	100.00
		otbn_mac_bignum_acc_err	21.000s	140.743us	5	5	100.00
		otbn_urnd_err	6.000s	10.207us	2	2	100.00
V2S	illegal_bus_access	otbn_illegal_mem_acc	8.000s	28.604us	5	5	100.00
V2S	otbn_mem_gnt_acc_err	otbn_mem_gnt_acc_err	10.000s	168.974us	1	2	50.00
V2S	otbn_non_sec_partial_wipe	otbn_partial_wipe	8.000s	92.157us	6	10	60.00
V2S	tl_intg_err	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
		otbn_tl_intg_err	31.000s	179.819us	20	20	100.00
V2S	passthru_mem_tl_intg_err	otbn_passthru_mem_tl_intg_err	35.000s	200.138us	20	20	100.00
V2S	prim_fsm_check	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	prim_count_check	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_mem_scramble	otbn_smoke	13.000s	67.988us	1	1	100.00
V2S	sec_cm_data_mem_integrity	otbn_dmem_err	15.000s	35.729us	15	15	100.00
V2S	sec_cm_instruction_mem_integrity	otbn_imem_err	16.000s	31.601us	10	10	100.00
V2S	sec_cm_bus_integrity	otbn_tl_intg_err	31.000s	179.819us	20	20	100.00
V2S	sec_cm_controller_fsm_global_esc	otbn_escalate	24.000s	48.341us	52	60	86.67
V2S	sec_cm_controller_fsm_local_esc	otbn_imem_err	16.000s	31.601us	10	10	100.00
		otbn_dmem_err	15.000s	35.729us	15	15	100.00
		otbn_zero_state_err_urnd	9.000s	275.377us	4	5	80.00
		otbn_illegal_mem_acc	8.000s	28.604us	5	5	100.00
		otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_controller_fsm_sparse	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_scramble_key_sideload	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_scramble_ctrl_fsm_local_esc	otbn_imem_err	16.000s	31.601us	10	10	100.00
		otbn_dmem_err	15.000s	35.729us	15	15	100.00
		otbn_zero_state_err_urnd	9.000s	275.377us	4	5	80.00
		otbn_illegal_mem_acc	8.000s	28.604us	5	5	100.00
		otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_scramble_ctrl_fsm_sparse	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_start_stop_ctrl_fsm_global_esc	otbn_escalate	24.000s	48.341us	52	60	86.67
V2S	sec_cm_start_stop_ctrl_fsm_local_esc	otbn_imem_err	16.000s	31.601us	10	10	100.00
		otbn_dmem_err	15.000s	35.729us	15	15	100.00
		otbn_zero_state_err_urnd	9.000s	275.377us	4	5	80.00
		otbn_illegal_mem_acc	8.000s	28.604us	5	5	100.00
		otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_start_stop_ctrl_fsm_sparse	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_data_reg_sw_sca	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_ctrl_redun	otbn_ctrl_redun	10.000s	27.088us	11	12	91.67
V2S	sec_cm_pc_ctrl_flow_redun	otbn_pc_ctrl_flow_redun	9.000s	22.518us	5	5	100.00
V2S	sec_cm_rnd_bus_consistency	otbn_rnd_sec_cm	1.567m	202.900us	5	5	100.00
V2S	sec_cm_rnd_rng_digest	otbn_rnd_sec_cm	1.567m	202.900us	5	5	100.00
V2S	sec_cm_rf_base_data_reg_sw_integrity	otbn_rf_base_intg_err	12.000s	53.673us	10	10	100.00
V2S	sec_cm_rf_base_data_reg_sw_glitch_detect	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_stack_wr_ptr_ctr_redun	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_rf_bignum_data_reg_sw_integrity	otbn_rf_bignum_intg_err	20.000s	70.744us	10	10	100.00
V2S	sec_cm_rf_bignum_data_reg_sw_glitch_detect	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_loop_stack_ctr_redun	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S	sec_cm_loop_stack_addr_integrity	otbn_stack_addr_integ_chk	10.000s	23.928us	4	5	80.00
V2S	sec_cm_call_stack_addr_integrity	otbn_stack_addr_integ_chk	10.000s	23.928us	4	5	80.00
V2S	sec_cm_start_stop_ctrl_state_consistency	otbn_sec_wipe_err	18.000s	55.297us	7	7	100.00
V2S	sec_cm_data_mem_sec_wipe	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_instruction_mem_sec_wipe	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_data_reg_sw_sec_wipe	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_write_mem_integrity	otbn_multi	1.867m	1.155ms	10	10	100.00
V2S	sec_cm_ctrl_flow_count	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_ctrl_flow_sca	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_data_mem_sw_noaccess	otbn_sw_no_acc	15.000s	92.973us	5	5	100.00
V2S	sec_cm_key_sideload	otbn_single	48.000s	192.512us	100	100	100.00
V2S	sec_cm_tlul_fifo_ctr_redun	otbn_sec_cm	8.050m	2.723ms	5	5	100.00
V2S		TOTAL			156	163	95.71
V3	stress_all_with_rand_reset	otbn_stress_all_with_rand_reset	13.333m	3.972ms	9	10	90.00
V3		TOTAL			9	10	90.00
		TOTAL			568	585	97.09

Testplan Progress

Items	Total	Written	Passing	Progress
V1	9	9	9	100.00
V2	11	11	9	81.82
V2S	20	20	16	80.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

Score	Block	Branch	Statement	Expression	Toggle	Fsm	Assertion	CoverGroup
98.87	99.60	95.44	99.69	93.43	92.37	97.44	91.38	99.16

Failure Buckets

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_sim_*/tb.sv,271): Assertion MatchingStatus_A has failed has 6 failures:

Test otbn_mem_gnt_acc_err has 1 failures.

0.otbn_mem_gnt_acc_err.59276991194534170808869630616528802179810290832529433950909324221737796590470
Line 288, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/0.otbn_mem_gnt_acc_err/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_sim_0.1/tb.sv,271): (time 14718661 PS) Assertion tb.MatchingStatus_A has failed
  UVM_ERROR @  14718661 ps: (tb.sv:271) [ASSERT FAILED] MatchingStatus_A
  UVM_INFO @  14718661 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test otbn_stack_addr_integ_chk has 1 failures.

0.otbn_stack_addr_integ_chk.76832464921565646864093082495672113443139194289707734159418143963804919978778
Line 289, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/0.otbn_stack_addr_integ_chk/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_sim_0.1/tb.sv,271): (time 21011083 PS) Assertion tb.MatchingStatus_A has failed
  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_env_0.1/otbn_insn_cnt_if.sv,21): (time 21011083 PS) Assertion tb.insn_cnt_if.InsnCntMatches_A has failed
  UVM_ERROR @  21011083 ps: (tb.sv:271) [ASSERT FAILED] MatchingStatus_A
  UVM_INFO @  21011083 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test otbn_escalate has 4 failures.

15.otbn_escalate.106555259155632190418717583374531000088223813502031796888895599513366803502825
Line 285, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/15.otbn_escalate/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_sim_0.1/tb.sv,271): (time 4520883 PS) Assertion tb.MatchingStatus_A has failed
  UVM_ERROR @   4520883 ps: (tb.sv:271) [ASSERT FAILED] MatchingStatus_A
  UVM_INFO @   4520883 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

38.otbn_escalate.51841457491004610280657823066175682273281121309668961857487960528952952105653
Line 286, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/38.otbn_escalate/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_sim_0.1/tb.sv,271): (time 2358230 PS) Assertion tb.MatchingStatus_A has failed
  UVM_ERROR @   2358230 ps: (tb.sv:271) [ASSERT FAILED] MatchingStatus_A
  UVM_INFO @   2358230 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_otbn_*/rtl/otbn.sv,884): Assertion NotBusyAndDone_A has failed has 4 failures:

Test otbn_partial_wipe has 4 failures.

1.otbn_partial_wipe.65568614443143205508803379002218401707225150650826983889190789265450726856299
Line 282, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/1.otbn_partial_wipe/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_otbn_0.1/rtl/otbn.sv,884): (time 6806742 PS) Assertion tb.dut.NotBusyAndDone_A has failed
  UVM_ERROR @   6806742 ps: (otbn.sv:884) [ASSERT FAILED] NotBusyAndDone_A
  UVM_INFO @   6806742 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.otbn_partial_wipe.62884259034939885853807036736129179275074044238781744690648426015896518477610
Line 288, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/3.otbn_partial_wipe/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_otbn_0.1/rtl/otbn.sv,884): (time 14273921 PS) Assertion tb.dut.NotBusyAndDone_A has failed
  UVM_ERROR @  14273921 ps: (otbn.sv:884) [ASSERT FAILED] NotBusyAndDone_A
  UVM_INFO @  14273921 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_model_agent_*/otbn_model_if.sv,137): Assertion NoModelErrs has failed has 3 failures:

Test otbn_escalate has 3 failures.

1.otbn_escalate.82146623852383968651081272202023736313388650125266335471438313347572029450089
Line 302, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/1.otbn_escalate/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_model_agent_0.1/otbn_model_if.sv,137): (time 11349704 PS) Assertion tb.model_if.NoModelErrs has failed
  UVM_ERROR @  11349704 ps: (otbn_model_if.sv:137) [ASSERT FAILED] NoModelErrs
  UVM_INFO @  11349704 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

12.otbn_escalate.30892028431380616438592126520456206424917744003756289306291918693408131412194
Line 302, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/12.otbn_escalate/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_model_agent_0.1/otbn_model_if.sv,137): (time 3002921 PS) Assertion tb.model_if.NoModelErrs has failed
  UVM_ERROR @   3002921 ps: (otbn_model_if.sv:137) [ASSERT FAILED] NoModelErrs
  UVM_INFO @   3002921 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_env_*/otbn_insn_cnt_if.sv,21): Assertion InsnCntMatches_A has failed has 2 failures:

Test otbn_zero_state_err_urnd has 1 failures.

2.otbn_zero_state_err_urnd.75745750469999644150382966074368735124348666123952771085365966661545714775066
Line 284, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/2.otbn_zero_state_err_urnd/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_env_0.1/otbn_insn_cnt_if.sv,21): (time 10589594 PS) Assertion tb.insn_cnt_if.InsnCntMatches_A has failed
  UVM_ERROR @  10589594 ps: (otbn_insn_cnt_if.sv:21) [ASSERT FAILED] InsnCntMatches_A
  UVM_INFO @  10589594 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test otbn_escalate has 1 failures.

7.otbn_escalate.43333961457266689962106027455304335398942842033701888992443370706055295029053
Line 287, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/7.otbn_escalate/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_dv_otbn_env_0.1/otbn_insn_cnt_if.sv,21): (time 48341066 PS) Assertion tb.insn_cnt_if.InsnCntMatches_A has failed
  UVM_ERROR @  48341066 ps: (otbn_insn_cnt_if.sv:21) [ASSERT FAILED] InsnCntMatches_A
  UVM_INFO @  48341066 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_vseq.sv:631) virtual_sequencer [otbn_dmem_err_vseq] expect alert:fatal to fire has 1 failures:
- Test otbn_stress_all_with_rand_reset has 1 failures.
  - 1.otbn_stress_all_with_rand_reset.36528395650305522662615029807718221790739038505789578546313971784535906329815
    Line 847, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/1.otbn_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 855573321 ps: (cip_base_vseq.sv:631) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.otbn_dmem_err_vseq] expect alert:fatal to fire
  UVM_INFO @ 855573321 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_otbn_*/rtl/otbn_core.sv,969): Assertion OnlyWriteLoadDataBignumWhenDMemValid_A has failed has 1 failures:
- Test otbn_ctrl_redun has 1 failures.
  - 3.otbn_ctrl_redun.10057859207641035756264759573428725121173720882609475039774048757311540209613
    Line 285, in log /container/opentitan-public/scratch/os_regression/otbn-sim-xcelium/3.otbn_ctrl_redun/latest/run.log
```
  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_otbn_0.1/rtl/otbn_core.sv,969): (time 15276804 PS) Assertion tb.dut.u_otbn_core.OnlyWriteLoadDataBignumWhenDMemValid_A has failed
  UVM_ERROR @  15276804 ps: (otbn_core.sv:969) [ASSERT FAILED] OnlyWriteLoadDataBignumWhenDMemValid_A
  UVM_INFO @  15276804 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```