KEYMGR Simulation Results

Saturday March 09 2024 20:02:36 UTC

GitHub Revision: `357d5d3c7b`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 3142200613

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	44.380s	4.963ms	50	50	100.00
V1	random	keymgr_random	2.292m	25.717ms	49	50	98.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.250s	160.709us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.530s	23.934us	17	20	85.00
V1	csr_bit_bash	keymgr_csr_bit_bash	16.970s	1.149ms	4	5	80.00
V1	csr_aliasing	keymgr_csr_aliasing	8.750s	134.959us	4	5	80.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.120s	205.784us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.530s	23.934us	17	20	85.00
		keymgr_csr_aliasing	8.750s	134.959us	4	5	80.00
V1		TOTAL			149	155	96.13
V2	cfgen_during_op	keymgr_cfg_regwen	2.529m	11.101ms	49	50	98.00
V2	sideload	keymgr_sideload	34.780s	3.963ms	50	50	100.00
		keymgr_sideload_kmac	1.170m	10.208ms	50	50	100.00
		keymgr_sideload_aes	1.543m	8.706ms	50	50	100.00
		keymgr_sideload_otbn	40.060s	2.188ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	29.970s	2.382ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	42.050s	3.010ms	49	50	98.00
V2	kmac_error_response	keymgr_kmac_rsp_err	2.193m	11.345ms	45	50	90.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.499m	9.396ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.349m	8.812ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	19.220s	692.257us	49	50	98.00
V2	stress_all	keymgr_stress_all	11.692m	128.649ms	50	50	100.00
V2	intr_test	keymgr_intr_test	1.030s	23.835us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.150s	319.024us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.740s	130.280us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.740s	130.280us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.250s	160.709us	5	5	100.00
		keymgr_csr_rw	1.530s	23.934us	17	20	85.00
		keymgr_csr_aliasing	8.750s	134.959us	4	5	80.00
		keymgr_same_csr_outstanding	2.520s	36.871us	12	20	60.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.250s	160.709us	5	5	100.00
		keymgr_csr_rw	1.530s	23.934us	17	20	85.00
		keymgr_csr_aliasing	8.750s	134.959us	4	5	80.00
		keymgr_same_csr_outstanding	2.520s	36.871us	12	20	60.00
V2		TOTAL			724	740	97.84
V2S	sec_cm_additional_check	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
		keymgr_tl_intg_err	17.250s	584.589us	14	20	70.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	49.610s	8.392ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	49.610s	8.392ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	49.610s	8.392ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	49.610s	8.392ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.270s	667.995us	14	20	70.00
V2S	prim_count_check	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	17.250s	584.589us	14	20	70.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	49.610s	8.392ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.529m	11.101ms	49	50	98.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	2.292m	25.717ms	49	50	98.00
		keymgr_csr_rw	1.530s	23.934us	17	20	85.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	2.292m	25.717ms	49	50	98.00
		keymgr_csr_rw	1.530s	23.934us	17	20	85.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	2.292m	25.717ms	49	50	98.00
		keymgr_csr_rw	1.530s	23.934us	17	20	85.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	42.050s	3.010ms	49	50	98.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.349m	8.812ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.349m	8.812ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	2.292m	25.717ms	49	50	98.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	25.170s	1.619ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	38.230s	1.244ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	42.050s	3.010ms	49	50	98.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	38.230s	1.244ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	38.230s	1.244ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	38.230s	1.244ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	40.480s	2.631ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	38.230s	1.244ms	50	50	100.00
V2S		TOTAL			153	165	92.73
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	14.990s	390.594us	46	50	92.00
V3		TOTAL			46	50	92.00
		TOTAL			1072	1110	96.58

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	3	42.86
V2	16	16	11	68.75
V2S	6	6	4	66.67
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.82	99.10	97.71	98.67	100.00	99.11	98.41	91.71

Failure Buckets

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 25 failures:

Test keymgr_csr_rw has 3 failures.

1.keymgr_csr_rw.1287676936
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[18] & 'hffffffff)))'
  UVM_ERROR @  18509234 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_5_rd_A
  UVM_INFO @  18509234 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

7.keymgr_csr_rw.3741752867
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[8] & 'hffffffff)))'
  UVM_ERROR @  12744657 ps: (keymgr_csr_assert_fpv.sv:381) [ASSERT FAILED] sealing_sw_binding_3_rd_A
  UVM_INFO @  12744657 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Test keymgr_csr_bit_bash has 1 failures.

1.keymgr_csr_bit_bash.1374300752
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[6] & 'hffffffff)))'
  UVM_ERROR @ 1148703023 ps: (keymgr_csr_assert_fpv.sv:371) [ASSERT FAILED] sealing_sw_binding_1_rd_A
  UVM_INFO @ 1148703023 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_shadow_reg_errors_with_csr_rw has 6 failures.

2.keymgr_shadow_reg_errors_with_csr_rw.2337170862
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[11] & 'hffffffff)))'
  UVM_ERROR @  26224074 ps: (keymgr_csr_assert_fpv.sv:396) [ASSERT FAILED] sealing_sw_binding_6_rd_A
  UVM_INFO @  26224074 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_shadow_reg_errors_with_csr_rw.2973163415
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[10] & 'hffffffff)))'
  UVM_ERROR @  93809369 ps: (keymgr_csr_assert_fpv.sv:391) [ASSERT FAILED] sealing_sw_binding_5_rd_A
  UVM_INFO @  93809369 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 4 more failures.

Test keymgr_csr_aliasing has 1 failures.

2.keymgr_csr_aliasing.3604308959
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_csr_aliasing/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[8] & 'hffffffff)))'
  UVM_ERROR @  69735779 ps: (keymgr_csr_assert_fpv.sv:381) [ASSERT FAILED] sealing_sw_binding_3_rd_A
  UVM_INFO @  69735779 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_same_csr_outstanding has 8 failures.

2.keymgr_same_csr_outstanding.2950465108
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[14] & 'hffffffff)))'
  UVM_ERROR @  72238758 ps: (keymgr_csr_assert_fpv.sv:411) [ASSERT FAILED] attest_sw_binding_1_rd_A
  UVM_INFO @  72238758 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_same_csr_outstanding.3055214993
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @  21441383 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @  21441383 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 6 more failures.

... and 1 more tests.

UVM_ERROR (keymgr_scoreboard.sv:1019) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 5 failures:

Test keymgr_kmac_rsp_err has 5 failures.

14.keymgr_kmac_rsp_err.3497871190
Line 417, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  33349732 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (316440404311327061297635222279119937965811401415309435595036037601091531344000468603074259601670676129662603049543874092450349670180215004194728136149481735715506066756346450273803659901615883222366093002077856026127332672371112443052942138317345048750352355665216693681099051506021041459165755889544184974724297994618266707905674433570718636402745123870141406997620617029209487660263979344013239196778510209942214922732643598449420701803352253311949477383175636425475545458617953056451266737086288244 [0x62c4c6260cde3ed95e90a41d8d7efcb125fc10556d94b648974cb25b6684156c3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f900ff4504556a3b379db1e3afe38b076cf34c49890d2304b1d365f0f5acc7d7af82ccf20f131c8260522267538b252a28c7838136a15bc1b8636caf49d19e03bb8a53d775fcf4f9a6631ae61ac01b6c869ec12888bdb7d2261526c742b8e7bf7eb9180649b5ac3d75cf2dddc07ca5bf43fc9a8978180c09babc1c3834c3e3d4c05afde9c1348d0b517b4a0a1b937a5174] vs 316440404311327061297635222279119937965811401415309435595036037601091531344000468603074259601670676129662603049543874092450349670180215004194728136149481735715506066756346450273803659901615883222366093002077856026127332672371112443052942138317345048750352355665216693681099051506021041459165755889544184974724297994618266707905674433570718636402745123870141406997620617029209487660263979344013239196778510209942214922732643598449420701803352253311949477383175636425475545458617953056451266737086288244 [0x62c4c6260cde3ed95e90a41d8d7efcb125fc10556d94b648974cb25b6684156c3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f900ff4504556a3b379db1e3afe38b076cf34c49890d2304b1d365f0f5acc7d7af82ccf20f131c8260522267538b252a28c7838136a15bc1b8636caf49d19e03bb8a53d775fcf4f9a6631ae61ac01b6c869ec12888bdb7d2261526c742b8e7bf7eb9180649b5ac3d75cf2dddc07ca5bf43fc9a8978180c09babc1c3834c3e3d4c05afde9c1348d0b517b4a0a1b937a5174]) cdi_type: Attestation
  
   DiversificationKey act: 0xfc9a8978180c09babc1c3834c3e3d4c05afde9c1348d0b517b4a0a1b937a5174, exp: 0xfc9a8978180c09babc1c3834c3e3d4c05afde9c1348d0b517b4a0a1b937a5174
   RomDigests act: 0xc7838136a15bc1b8636caf49d19e03bb8a53d775fcf4f9a6631ae61ac01b6c869ec12888bdb7d2261526c742b8e7bf7eb9180649b5ac3d75cf2dddc07ca5bf43, exp: 0xc7838136a15bc1b8636caf49d19e03bb8a53d775fcf4f9a6631ae61ac01b6c869ec12888bdb7d2261526c742b8e7bf7eb9180649b5ac3d75cf2dddc07ca5bf43
   HealthMeasurement act: 0x82ccf20f131c8260522267538b252a28, exp: 0x82ccf20f131c8260522267538b252a28

16.keymgr_kmac_rsp_err.2456999406
Line 319, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  17464216 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (738666444949878320503481382354669851279676056597731677683085474115763346119814816328989589414971432336658046961752712617265027064194516115679821161228942929949954142451626866661049870865220236419152181260781611335717527591029740020635765094505792153964765687122507294820458953824626621539471469687052229594007530710902671480130587628255930371457764782078741398385788455897050722356624167039619756103815632346876340756567981105503195887970571659353657771921330844324822712812019161189288690011371858808 [0xe68e377dabf425552f4949b895d8b275507a2e5615eaa3a07d2c39765c0bdac43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 738666444949878320503481382354669851279676056597731677683085474115763346119814816328989589414971432336658046961752712617265027064194516115679821161228942929949954142451626866661049870865220236419152181260781611335717527591029740020635765094505792153964765687122507294820458953824626621539471469687052229594007530710902671480130587628255930371457764782078741398385788455897050722356624167039619756103815632346876340756567981105503195887970571659353657771921330844324822712812019161189288690011371858808 [0xe68e377dabf425552f4949b895d8b275507a2e5615eaa3a07d2c39765c0bdac43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigests act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

... and 3 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

16.keymgr_stress_all_with_rand_reset.111670336
Line 1746, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 800261035 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 800261035 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

27.keymgr_stress_all_with_rand_reset.2185599307
Line 569, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/27.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 642346917 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 642346917 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

36.keymgr_stress_all_with_rand_reset.3975107292
Line 515, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 183990806 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2346149009 [0x8bd76491] vs 2346149009 [0x8bd76491])
  UVM_INFO @ 183990806 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

37.keymgr_stress_all_with_rand_reset.830022033
Line 549, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/37.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @  59405955 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (765143448 [0x2d9b2998] vs 765143448 [0x2d9b2998])
  UVM_INFO @  59405955 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 1 failures:
- Test keymgr_random has 1 failures.
  - 6.keymgr_random.3640926561
    Line 273, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_random/latest/run.log
```
  UVM_ERROR @  10442087 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  10442087 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:241) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 13.keymgr_sync_async_fault_cross.167488037
    Line 365, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/13.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 234837862 ps: (cip_base_scoreboard.sv:241) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 234837862 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.start` has 1 failures:
- Test keymgr_cfg_regwen has 1 failures.
  - 25.keymgr_cfg_regwen.4149078168
    Line 459, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/25.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @  16409466 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (0 [0x0] vs 1 [0x1]) reg name: keymgr_reg_block.start
  UVM_INFO @  16409466 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (csr_utils_pkg.sv:566) [csr_utils::csr_spinwait] timeout keymgr_reg_block.working_state (addr=*) == * has 1 failures:
- Test keymgr_lc_disable has 1 failures.
  - 26.keymgr_lc_disable.2632907416
    Line 565, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/26.keymgr_lc_disable/latest/run.log
```
  UVM_FATAL @ 10294154540 ps: (csr_utils_pkg.sv:566) [csr_utils::csr_spinwait] timeout keymgr_reg_block.working_state (addr=0xb0c663e8) == 0x5
  UVM_INFO @ 10294154540 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```