KEYMGR Simulation Results

Sunday May 28 2023 07:05:15 UTC

GitHub Revision: `c06cc3921`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 2869101736

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.065m	3.826ms	50	50	100.00
V1	random	keymgr_random	56.720s	5.376ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.520s	129.736us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.560s	107.384us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	15.180s	1.047ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	8.670s	248.500us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.900s	95.009us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.560s	107.384us	20	20	100.00
		keymgr_csr_aliasing	8.670s	248.500us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	44.200s	5.320ms	50	50	100.00
V2	sideload	keymgr_sideload	53.590s	6.121ms	50	50	100.00
		keymgr_sideload_kmac	46.760s	2.150ms	50	50	100.00
		keymgr_sideload_aes	1.481m	10.884ms	50	50	100.00
		keymgr_sideload_otbn	55.470s	1.761ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	20.790s	3.703ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	7.680s	946.139us	49	50	98.00
V2	kmac_error_response	keymgr_kmac_rsp_err	38.590s	4.963ms	41	50	82.00
V2	invalid_sw_input	keymgr_sw_invalid_input	41.140s	7.837ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.074m	9.541ms	49	50	98.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	24.590s	1.538ms	49	50	98.00
V2	stress_all	keymgr_stress_all	10.940m	71.085ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.890s	22.644us	50	50	100.00
V2	alert_test	keymgr_alert_test	0.990s	45.993us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.710s	611.186us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.710s	611.186us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.520s	129.736us	5	5	100.00
		keymgr_csr_rw	1.560s	107.384us	20	20	100.00
		keymgr_csr_aliasing	8.670s	248.500us	5	5	100.00
		keymgr_same_csr_outstanding	3.880s	113.618us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.520s	129.736us	5	5	100.00
		keymgr_csr_rw	1.560s	107.384us	20	20	100.00
		keymgr_csr_aliasing	8.670s	248.500us	5	5	100.00
		keymgr_same_csr_outstanding	3.880s	113.618us	20	20	100.00
V2		TOTAL			728	740	98.38
V2S	sec_cm_additional_check	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
		keymgr_tl_intg_err	38.770s	6.466ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	7.820s	1.580ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	7.820s	1.580ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	7.820s	1.580ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	7.820s	1.580ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	21.100s	2.206ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	38.770s	6.466ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	7.820s	1.580ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	44.200s	5.320ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	56.720s	5.376ms	50	50	100.00
		keymgr_csr_rw	1.560s	107.384us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	56.720s	5.376ms	50	50	100.00
		keymgr_csr_rw	1.560s	107.384us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	56.720s	5.376ms	50	50	100.00
		keymgr_csr_rw	1.560s	107.384us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	7.680s	946.139us	49	50	98.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.074m	9.541ms	49	50	98.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.074m	9.541ms	49	50	98.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	56.720s	5.376ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	27.380s	2.729ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	13.890s	1.316ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	7.680s	946.139us	49	50	98.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	13.890s	1.316ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	13.890s	1.316ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	13.890s	1.316ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	2.717m	68.063ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	13.890s	1.316ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	19.570s	328.401us	48	50	96.00
V3		TOTAL			48	50	96.00
		TOTAL			1096	1110	98.74

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	12	75.00
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.47	99.09	97.96	98.37	97.67	98.99	98.38	91.78

Failure Buckets

UVM_ERROR (keymgr_scoreboard.sv:1012) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 9 failures:

Test keymgr_kmac_rsp_err has 9 failures.

2.keymgr_kmac_rsp_err.1601289551
Line 286, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  50420343 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (6275084268236572682240591202310296692942711569965768157139789315472080405362276903932516096657749983287418350842581768537522323614870911283638822518313529255130229721097207732322287715434985458478789477830693941852364555797632323453304533293159739648415792039960597196162110182279705953063708066740255334771949151773932885098469853042362581611936107881874068532142314404497894583488744108832330296996974753426424289848547738 [0xe2ca78b4c9dab51003f3018ee84e428284135720746854043e2a3fed2a22ef983a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ab58d5a61a8c57f98d41672a29216288024a4616f2a3c9fde2b1d0b58f45aa3b2c153f65fca06c4ba5e49660e462e05426f910f4a8744f66c0f1dc6f40fc9a8b0c5b84c316ab5b95247f18d60e3aa0bc236a9612f2f8d4ab54be5d1fd2b7ae66194e2a585e53162eb109260f65e9599a] vs 6275084268236572682240591202310296692942711569965768157139789315472080405362276903932516096657749983287418350842581768537522323614870911283638822518313529255130229721097207732322287715434985458478789477830693941852364555797632323453304533293159739648415792039960597196162110182279705953063708066740255334771949151773932885098469853042362581611936107881874068532142314404497894583488744108832330296996974753426424289848547738 [0xe2ca78b4c9dab51003f3018ee84e428284135720746854043e2a3fed2a22ef983a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ab58d5a61a8c57f98d41672a29216288024a4616f2a3c9fde2b1d0b58f45aa3b2c153f65fca06c4ba5e49660e462e05426f910f4a8744f66c0f1dc6f40fc9a8b0c5b84c316ab5b95247f18d60e3aa0bc236a9612f2f8d4ab54be5d1fd2b7ae66194e2a585e53162eb109260f65e9599a]) cdi_type: Attestation
  
   DiversificationKey act: 0x236a9612f2f8d4ab54be5d1fd2b7ae66194e2a585e53162eb109260f65e9599a, exp: 0x236a9612f2f8d4ab54be5d1fd2b7ae66194e2a585e53162eb109260f65e9599a
   RomDigest act: 0x26f910f4a8744f66c0f1dc6f40fc9a8b0c5b84c316ab5b95247f18d60e3aa0bc, exp: 0x26f910f4a8744f66c0f1dc6f40fc9a8b0c5b84c316ab5b95247f18d60e3aa0bc
   HealthMeasurement act: 0x2c153f65fca06c4ba5e49660e462e054, exp: 0x2c153f65fca06c4ba5e49660e462e054

6.keymgr_kmac_rsp_err.1056976536
Line 300, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  10024588 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (2608029023364290243163563808795611827834889062109487552195754956945184802007983914002736845734590081644110041482612127503205950578928184598193572428568675508480282861678581687172033851514003598245385915507290869996426339197882645441989722266142393781757115106600974006543535393137410426307520332114099771672366878160711414918421758322597753868043291427069897214138851573300998104478323434624277557356802379473851105494722298 [0x5e421021042c9a05808871a1dc64149b1073e93f652adb6759ac443cac3098143a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f95d2625ab87c3e084d729e4b6988fd201590961c8c32434e56a69534064ea9d34d16f07e02c69d081773fcde73bb1fe39d24cc43c4d192bbadd7ed0ad411a88771cebaf971b8ed1ccab950df284eb132fc5f0d54190c8a51cef171c9e803a34143011795250bb967f90a2315e11bc66fa] vs 2608029023364290243163563808795611827834889062109487552195754956945184802007983914002736845734590081644110041482612127503205950578928184598193572428568675508480282861678581687172033851514003598245385915507290869996426339197882645441989722266142393781757115106600974006543535393137410426307520332114099771672366878160711414918421758322597753868043291427069897214138851573300998104478323434624277557356802379473851105494722298 [0x5e421021042c9a05808871a1dc64149b1073e93f652adb6759ac443cac3098143a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f95d2625ab87c3e084d729e4b6988fd201590961c8c32434e56a69534064ea9d34d16f07e02c69d081773fcde73bb1fe39d24cc43c4d192bbadd7ed0ad411a88771cebaf971b8ed1ccab950df284eb132fc5f0d54190c8a51cef171c9e803a34143011795250bb967f90a2315e11bc66fa]) cdi_type: Attestation
  
   DiversificationKey act: 0xc5f0d54190c8a51cef171c9e803a34143011795250bb967f90a2315e11bc66fa, exp: 0xc5f0d54190c8a51cef171c9e803a34143011795250bb967f90a2315e11bc66fa
   RomDigest act: 0xd24cc43c4d192bbadd7ed0ad411a88771cebaf971b8ed1ccab950df284eb132f, exp: 0xd24cc43c4d192bbadd7ed0ad411a88771cebaf971b8ed1ccab950df284eb132f
   HealthMeasurement act: 0xd16f07e02c69d081773fcde73bb1fe39, exp: 0xd16f07e02c69d081773fcde73bb1fe39

... and 7 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

24.keymgr_stress_all_with_rand_reset.4040342893
Line 827, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 620701236 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 620701236 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

38.keymgr_stress_all_with_rand_reset.683167909
Line 1130, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/38.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 286156331 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 286156331 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_scoreboard.sv:241) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 14.keymgr_sync_async_fault_cross.3154100535
    Line 314, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 398315449 ps: (cip_base_scoreboard.sv:241) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 398315449 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:633) [scoreboard] Check failed act_state == addr_phase_working_state (* [*] vs * [*]) has 1 failures:
- Test keymgr_lc_disable has 1 failures.
  - 15.keymgr_lc_disable.2705706195
    Line 333, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/15.keymgr_lc_disable/latest/run.log
```
  UVM_ERROR @  14039188 ps: (keymgr_scoreboard.sv:633) [uvm_test_top.env.scoreboard] Check failed act_state == addr_phase_working_state (1 [0x1] vs 6 [0x6])
  UVM_INFO @  14039188 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 1 failures:
- Test keymgr_hwsw_invalid_input has 1 failures.
  - 47.keymgr_hwsw_invalid_input.1681619691
    Line 247, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/47.keymgr_hwsw_invalid_input/latest/run.log
```
  UVM_ERROR @   6274332 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   6274332 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```