KEYMGR Simulation Results

Saturday May 04 2024 19:02:32 UTC

GitHub Revision: `f4962c85dd`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 1307141976

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.144m	3.192ms	50	50	100.00
V1	random	keymgr_random	1.594m	9.354ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.130s	61.862us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.310s	455.626us	16	20	80.00
V1	csr_bit_bash	keymgr_csr_bit_bash	14.260s	511.118us	3	5	60.00
V1	csr_aliasing	keymgr_csr_aliasing	13.850s	357.394us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.610s	37.542us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.310s	455.626us	16	20	80.00
		keymgr_csr_aliasing	13.850s	357.394us	5	5	100.00
V1		TOTAL			149	155	96.13
V2	cfgen_during_op	keymgr_cfg_regwen	1.357m	6.312ms	49	50	98.00
V2	sideload	keymgr_sideload	1.433m	4.325ms	50	50	100.00
		keymgr_sideload_kmac	1.433m	4.442ms	50	50	100.00
		keymgr_sideload_aes	55.740s	9.158ms	50	50	100.00
		keymgr_sideload_otbn	1.214m	22.439ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	28.260s	2.790ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	13.290s	1.243ms	48	50	96.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.567m	11.310ms	42	50	84.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.161m	2.122ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	39.480s	1.279ms	49	50	98.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	32.740s	3.745ms	49	50	98.00
V2	stress_all	keymgr_stress_all	4.016m	37.997ms	48	50	96.00
V2	intr_test	keymgr_intr_test	1.020s	18.039us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.190s	109.140us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.460s	437.619us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.460s	437.619us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.130s	61.862us	5	5	100.00
		keymgr_csr_rw	1.310s	455.626us	16	20	80.00
		keymgr_csr_aliasing	13.850s	357.394us	5	5	100.00
		keymgr_same_csr_outstanding	3.930s	89.483us	17	20	85.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.130s	61.862us	5	5	100.00
		keymgr_csr_rw	1.310s	455.626us	16	20	80.00
		keymgr_csr_aliasing	13.850s	357.394us	5	5	100.00
		keymgr_same_csr_outstanding	3.930s	89.483us	17	20	85.00
V2		TOTAL			722	740	97.57
V2S	sec_cm_additional_check	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
		keymgr_tl_intg_err	33.670s	1.511ms	15	20	75.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	1.128m	4.596ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	1.128m	4.596ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	1.128m	4.596ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	1.128m	4.596ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	14.940s	2.272ms	15	20	75.00
V2S	prim_count_check	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	33.670s	1.511ms	15	20	75.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	1.128m	4.596ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.357m	6.312ms	49	50	98.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.594m	9.354ms	50	50	100.00
		keymgr_csr_rw	1.310s	455.626us	16	20	80.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.594m	9.354ms	50	50	100.00
		keymgr_csr_rw	1.310s	455.626us	16	20	80.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.594m	9.354ms	50	50	100.00
		keymgr_csr_rw	1.310s	455.626us	16	20	80.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	13.290s	1.243ms	48	50	96.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	39.480s	1.279ms	49	50	98.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	39.480s	1.279ms	49	50	98.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.594m	9.354ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	45.680s	4.449ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	1.166m	4.119ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	13.290s	1.243ms	48	50	96.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	1.166m	4.119ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	1.166m	4.119ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	1.166m	4.119ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	50.660s	10.259ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	1.166m	4.119ms	50	50	100.00
V2S		TOTAL			155	165	93.94
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	13.270s	191.707us	47	50	94.00
V3		TOTAL			47	50	94.00
		TOTAL			1073	1110	96.67

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	5	71.43
V2	16	16	9	56.25
V2S	6	6	4	66.67
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.85	99.10	97.99	98.74	100.00	99.11	98.41	91.58

Failure Buckets

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 19 failures:

Test keymgr_csr_rw has 4 failures.

0.keymgr_csr_rw.3471954390
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @  19375567 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @  19375567 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_csr_rw.2705586525
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[6] & 'hffffffff)))'
  UVM_ERROR @  10804665 ps: (keymgr_csr_assert_fpv.sv:371) [ASSERT FAILED] sealing_sw_binding_1_rd_A
  UVM_INFO @  10804665 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_csr_bit_bash has 2 failures.

1.keymgr_csr_bit_bash.2590895722
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[16] & 'hffffffff)))'
  UVM_ERROR @ 546419446 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] attest_sw_binding_3_rd_A
  UVM_INFO @ 546419446 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

4.keymgr_csr_bit_bash.913220872
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @ 871082339 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @ 871082339 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_same_csr_outstanding has 3 failures.

1.keymgr_same_csr_outstanding.3576375941
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[18] & 'hffffffff)))'
  UVM_ERROR @ 113466421 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_5_rd_A
  UVM_INFO @ 113466421 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

6.keymgr_same_csr_outstanding.3244559732
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[8] & 'hffffffff)))'
  UVM_ERROR @   7151113 ps: (keymgr_csr_assert_fpv.sv:381) [ASSERT FAILED] sealing_sw_binding_3_rd_A
  UVM_INFO @   7151113 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Test keymgr_shadow_reg_errors_with_csr_rw has 5 failures.

2.keymgr_shadow_reg_errors_with_csr_rw.1449571767
Line 260, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[14] & 'hffffffff)))'
  UVM_ERROR @ 126784118 ps: (keymgr_csr_assert_fpv.sv:411) [ASSERT FAILED] attest_sw_binding_1_rd_A
  UVM_INFO @ 126784118 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_shadow_reg_errors_with_csr_rw.2652493352
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[18] & 'hffffffff)))'
  UVM_ERROR @  58664421 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_5_rd_A
  UVM_INFO @  58664421 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 3 more failures.

Test keymgr_tl_intg_err has 5 failures.

2.keymgr_tl_intg_err.946832511
Line 258, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[20] & 'hffffffff)))'
  UVM_ERROR @  46624572 ps: (keymgr_csr_assert_fpv.sv:441) [ASSERT FAILED] attest_sw_binding_7_rd_A
  UVM_INFO @  46624572 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_tl_intg_err.1205517387
Line 262, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @  44875410 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @  44875410 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 3 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1019) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 8 failures:

Test keymgr_kmac_rsp_err has 8 failures.

2.keymgr_kmac_rsp_err.1787685470
Line 382, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  43018834 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (194417771838970232564935412852606804042140679320097919842310834852353256407505586717127407991129893810259713545799343580216544552208713373343575658635704918390187916255202891350553352075479055228770407929937476404781686271728154859222883528989228842904262524793530328444547060791798634781026195013891025245070039383521055670902449013441964485322917695727328797269509502969091351775448017309014207301310917517457651037767613118211396461535397186004330494839595769911029842708500026189492403611530554232 [0x3caeb55da848b55fb8c6fe60000000008010d248839a85e796bd928f0b9595e13a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 194417771838970232564935412852606804042140679320097919842310834852353256407505586717127407991129893810259713545799343580216544552208713373343575658635704918390187916255202891350553352075479055228770407929937476404781686271728154859222883528989228842904262524793530328444547060791798634781026195013891025245070039383521055670902449013441964485322917695727328797269509502969091351775448017309014207301310917517457651037767613118211396461535397186004330494839595769911029842708500026189492403611530554232 [0x3caeb55da848b55fb8c6fe60000000008010d248839a85e796bd928f0b9595e13a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigests act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

9.keymgr_kmac_rsp_err.1592245446
Line 274, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/9.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  73467055 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (1605930754565491250042339328357387143542526556207941497641627338815315533913352297993149127478058689648893632913198171530248200852941984195549381886973422681255015032114135203137537205090915768177155400739687910522573101873379020998428869845811812807166779785114651591778191252731128191543260763372077924040897239573632965149663302566954135612592731018052412953178136996315584526597000297796699302924983250572710026026541509 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9a4155d232437fc2959bd4508daa1113807b8180fbfa964c7de8cd4ce84f314d423e1584fff29db5be5c5ffa421b9ac4a871bb27ae6cd234705adf4014775678744635c497a55680b760489869a3cf79b209ea4d677fa0d12c1096d688cc7f15c3a3b94c306e9bb51d4f754b7f59200f31fb7b456b45da63e37feb1237e9c120350689a5392d241b6106ffbc6d8a5e1c5] vs 1605930754565491250042339328357387143542526556207941497641627338815315533913352297993149127478058689648893632913198171530248200852941984195549381886973422681255015032114135203137537205090915768177155400739687910522573101873379020998428869845811812807166779785114651591778191252731128191543260763372077924040897239573632965149663302566954135612592731018052412953178136996315584526597000297796699302924983250572710026026541509 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9a4155d232437fc2959bd4508daa1113807b8180fbfa964c7de8cd4ce84f314d423e1584fff29db5be5c5ffa421b9ac4a871bb27ae6cd234705adf4014775678744635c497a55680b760489869a3cf79b209ea4d677fa0d12c1096d688cc7f15c3a3b94c306e9bb51d4f754b7f59200f31fb7b456b45da63e37feb1237e9c120350689a5392d241b6106ffbc6d8a5e1c5]) cdi_type: Attestation
  
   DiversificationKey act: 0x1fb7b456b45da63e37feb1237e9c120350689a5392d241b6106ffbc6d8a5e1c5, exp: 0x1fb7b456b45da63e37feb1237e9c120350689a5392d241b6106ffbc6d8a5e1c5
   RomDigests act: 0x871bb27ae6cd234705adf4014775678744635c497a55680b760489869a3cf79b209ea4d677fa0d12c1096d688cc7f15c3a3b94c306e9bb51d4f754b7f59200f3, exp: 0x871bb27ae6cd234705adf4014775678744635c497a55680b760489869a3cf79b209ea4d677fa0d12c1096d688cc7f15c3a3b94c306e9bb51d4f754b7f59200f3
   HealthMeasurement act: 0x23e1584fff29db5be5c5ffa421b9ac4a, exp: 0x23e1584fff29db5be5c5ffa421b9ac4a

... and 6 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 3 failures:

Test keymgr_stress_all_with_rand_reset has 3 failures.

6.keymgr_stress_all_with_rand_reset.3696669292
Line 1051, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 138638844 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 138638844 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

20.keymgr_stress_all_with_rand_reset.319048659
Line 1356, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/20.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 157742413 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 157742413 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share0_output_*` has 2 failures:

Test keymgr_lc_disable has 1 failures.

24.keymgr_lc_disable.4234259711
Line 354, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_lc_disable/latest/run.log

  UVM_ERROR @  46225555 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (3047449961 [0xb5a46569] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share0_output_6
  UVM_INFO @  46225555 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 1 failures.

27.keymgr_stress_all.1115816881
Line 1265, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/27.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 516454459 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (1967544691 [0x75465973] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share0_output_1
  UVM_INFO @ 516454459 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 1 failures:
- Test keymgr_lc_disable has 1 failures.
  - 4.keymgr_lc_disable.549443752
    Line 273, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_lc_disable/latest/run.log
```
  UVM_ERROR @   7773294 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (1184890882 [0x46a00002] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_1
  UVM_INFO @   7773294 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.start` has 1 failures:
- Test keymgr_cfg_regwen has 1 failures.
  - 23.keymgr_cfg_regwen.467574533
    Line 305, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/23.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @   4074833 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (0 [0x0] vs 1 [0x1]) reg name: keymgr_reg_block.start
  UVM_INFO @   4074833 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 1 failures:
- Test keymgr_hwsw_invalid_input has 1 failures.
  - 25.keymgr_hwsw_invalid_input.1280553452
    Line 701, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/25.keymgr_hwsw_invalid_input/latest/run.log
```
  UVM_ERROR @  73695950 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  73695950 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:633) [scoreboard] Check failed act_state == addr_phase_working_state (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 28.keymgr_stress_all.1922438391
    Line 1091, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/28.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 417513019 ps: (keymgr_scoreboard.sv:633) [uvm_test_top.env.scoreboard] Check failed act_state == addr_phase_working_state (1 [0x1] vs 6 [0x6])
  UVM_INFO @ 417513019 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:241) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 36.keymgr_sync_async_fault_cross.591096351
    Line 373, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 100213109 ps: (cip_base_scoreboard.sv:241) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 100213109 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```