KEYMGR Simulation Results

Monday September 09 2024 02:20:26 UTC

GitHub Revision: `af2d1709f9`

Branch: os_regression_2024_09_08

Simulator: VCS

Build randomization enabled with --build-seed 13980492992314588037778262839223440914483141513139750793389284041724730149540

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.001m	1.600ms	50	50	100.00
V1	random	keymgr_random	1.048m	2.855ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	2.290s	36.318us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	2.050s	33.442us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	18.280s	9.558ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	14.740s	382.449us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	3.230s	30.989us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	2.050s	33.442us	20	20	100.00
		keymgr_csr_aliasing	14.740s	382.449us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	1.866m	2.278ms	50	50	100.00
V2	sideload	keymgr_sideload	45.500s	1.460ms	50	50	100.00
		keymgr_sideload_kmac	54.960s	22.661ms	50	50	100.00
		keymgr_sideload_aes	51.330s	1.775ms	50	50	100.00
		keymgr_sideload_otbn	35.600s	3.467ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	26.440s	849.220us	50	50	100.00
V2	lc_disable	keymgr_lc_disable	7.570s	123.606us	49	50	98.00
V2	kmac_error_response	keymgr_kmac_rsp_err	12.250s	9.465ms	50	50	100.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.185m	35.384ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	35.340s	1.198ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	25.590s	13.046ms	50	50	100.00
V2	stress_all	keymgr_stress_all	3.285m	6.702ms	49	50	98.00
V2	intr_test	keymgr_intr_test	1.220s	22.085us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.490s	19.513us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	6.140s	2.190ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	6.140s	2.190ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	2.290s	36.318us	5	5	100.00
		keymgr_csr_rw	2.050s	33.442us	20	20	100.00
		keymgr_csr_aliasing	14.740s	382.449us	5	5	100.00
		keymgr_same_csr_outstanding	5.130s	431.256us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	2.290s	36.318us	5	5	100.00
		keymgr_csr_rw	2.050s	33.442us	20	20	100.00
		keymgr_csr_aliasing	14.740s	382.449us	5	5	100.00
		keymgr_same_csr_outstanding	5.130s	431.256us	20	20	100.00
V2		TOTAL			738	740	99.73
V2S	sec_cm_additional_check	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
		keymgr_tl_intg_err	10.600s	212.910us	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	5.570s	155.395us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	5.570s	155.395us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	5.570s	155.395us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	5.570s	155.395us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.860s	1.069ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	10.600s	212.910us	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	5.570s	155.395us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.866m	2.278ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.048m	2.855ms	50	50	100.00
		keymgr_csr_rw	2.050s	33.442us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.048m	2.855ms	50	50	100.00
		keymgr_csr_rw	2.050s	33.442us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.048m	2.855ms	50	50	100.00
		keymgr_csr_rw	2.050s	33.442us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	7.570s	123.606us	49	50	98.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	35.340s	1.198ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	35.340s	1.198ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.048m	2.855ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	19.560s	923.911us	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	21.650s	4.378ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	7.570s	123.606us	49	50	98.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	21.650s	4.378ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	21.650s	4.378ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	21.650s	4.378ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	15.170s	1.402ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	21.650s	4.378ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	26.990s	3.149ms	30	50	60.00
V3		TOTAL			30	50	60.00
		TOTAL			1088	1110	98.02

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	14	87.50
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.80	99.04	98.19	98.53	100.00	99.02	98.63	91.22

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:867) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 18 failures:
- Test keymgr_stress_all_with_rand_reset has 18 failures.
  - 0.keymgr_stress_all_with_rand_reset.13962656846188985771205215899802390321717191498788959921825240891495192235392
    Line 691, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/0.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 247789620 ps: (cip_base_vseq.sv:867) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10002 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 247789620 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 1.keymgr_stress_all_with_rand_reset.44462911973513297095380543712405721783365503064574517050598963382412297875157
    Line 548, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/1.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1020547944 ps: (cip_base_vseq.sv:867) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 1020547944 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 16 more failures.
UVM_ERROR (cip_base_scoreboard.sv:304) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 2 failures:
- Test keymgr_stress_all has 1 failures.
  - 15.keymgr_stress_all.44123428542204961837059248596883030642930607150023426323809215673557082460819
    Line 196, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/15.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @  11084320 ps: (cip_base_scoreboard.sv:304) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  11084320 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 35.keymgr_stress_all_with_rand_reset.34952892554496966673516424391371941433517479040787732079718823970188254332078
    Line 1193, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/35.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 297664716 ps: (cip_base_scoreboard.sv:304) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 297664716 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

UVM_ERROR (keymgr_if.sv:557) [keymgr_if] Check failed {act_key.key[*], act_key.key[*]} !== keys_a_array[state][cdi][dest] (* [*] vs * [*]) AES key at state StCreatorRootKey for Attestation Aes has 1 failures:

Test keymgr_lc_disable has 1 failures.

5.keymgr_lc_disable.10946033666644270273284606981348567088063662156514895318919662770521791010489
Line 280, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/5.keymgr_lc_disable/latest/run.log

  UVM_ERROR @ 1797528171 ps: (keymgr_if.sv:557) [keymgr_if] Check failed {act_key.key[1], act_key.key[0]} !== keys_a_array[state][cdi][dest] (12214102276264232454992624976328088264738337792148866795944144677133236500531112971723512154977363818423331488717514166499652207221712057258837939899915761 [0xe93549ce6c61e05a8f1cda9147fb2cb512d5d07aa5739926412dd3103290827216f27c26f3874ccc0a51b0b82908e1a6815dae8cda2b9fbffb04f4ec2d64ddf1] vs 12214102276264232454992624976328088264738337792148866795944144677133236500531112971723512154977363818423331488717514166499652207221712057258837939899915761 [0xe93549ce6c61e05a8f1cda9147fb2cb512d5d07aa5739926412dd3103290827216f27c26f3874ccc0a51b0b82908e1a6815dae8cda2b9fbffb04f4ec2d64ddf1]) AES key at state StCreatorRootKey for Attestation Aes
  UVM_INFO @ 1797528171 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:692) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 17.keymgr_stress_all_with_rand_reset.51479960588371561151263346009957762540189638481718253755041028955074466690935
    Line 371, in log /workspaces/repo/scratch/os_regression_2024_09_08/keymgr-sim-vcs/17.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 382430334 ps: (keymgr_scoreboard.sv:692) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 382430334 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```