KEYMGR Simulation Results

Saturday March 23 2024 19:02:16 UTC

GitHub Revision: `d638842535`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 2621884165

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.208m	14.210ms	50	50	100.00
V1	random	keymgr_random	1.244m	10.084ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.640s	368.036us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.610s	255.648us	16	20	80.00
V1	csr_bit_bash	keymgr_csr_bit_bash	14.890s	563.399us	1	5	20.00
V1	csr_aliasing	keymgr_csr_aliasing	14.170s	1.231ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.020s	29.265us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.610s	255.648us	16	20	80.00
		keymgr_csr_aliasing	14.170s	1.231ms	5	5	100.00
V1		TOTAL			147	155	94.84
V2	cfgen_during_op	keymgr_cfg_regwen	2.924m	7.080ms	50	50	100.00
V2	sideload	keymgr_sideload	1.405m	4.493ms	50	50	100.00
		keymgr_sideload_kmac	53.260s	3.855ms	50	50	100.00
		keymgr_sideload_aes	1.359m	8.407ms	50	50	100.00
		keymgr_sideload_otbn	53.030s	2.322ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	1.028m	4.871ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	10.650s	384.928us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.186m	6.040ms	48	50	96.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.378m	7.489ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.267m	11.552ms	49	50	98.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	26.040s	1.342ms	49	50	98.00
V2	stress_all	keymgr_stress_all	6.638m	52.398ms	46	50	92.00
V2	intr_test	keymgr_intr_test	0.940s	66.983us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.080s	230.086us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.410s	770.109us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.410s	770.109us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.640s	368.036us	5	5	100.00
		keymgr_csr_rw	1.610s	255.648us	16	20	80.00
		keymgr_csr_aliasing	14.170s	1.231ms	5	5	100.00
		keymgr_same_csr_outstanding	3.500s	92.795us	18	20	90.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.640s	368.036us	5	5	100.00
		keymgr_csr_rw	1.610s	255.648us	16	20	80.00
		keymgr_csr_aliasing	14.170s	1.231ms	5	5	100.00
		keymgr_same_csr_outstanding	3.500s	92.795us	18	20	90.00
V2		TOTAL			730	740	98.65
V2S	sec_cm_additional_check	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
		keymgr_tl_intg_err	1.014m	6.608ms	16	20	80.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	12.140s	580.494us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	12.140s	580.494us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	12.140s	580.494us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	12.140s	580.494us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	17.770s	524.463us	16	20	80.00
V2S	prim_count_check	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	1.014m	6.608ms	16	20	80.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	12.140s	580.494us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.924m	7.080ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.244m	10.084ms	50	50	100.00
		keymgr_csr_rw	1.610s	255.648us	16	20	80.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.244m	10.084ms	50	50	100.00
		keymgr_csr_rw	1.610s	255.648us	16	20	80.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.244m	10.084ms	50	50	100.00
		keymgr_csr_rw	1.610s	255.648us	16	20	80.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	10.650s	384.928us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.267m	11.552ms	49	50	98.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.267m	11.552ms	49	50	98.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.244m	10.084ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	37.180s	1.371ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	27.950s	1.066ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	10.650s	384.928us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	27.950s	1.066ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	27.950s	1.066ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	27.950s	1.066ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	27.800s	1.202ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	27.950s	1.066ms	50	50	100.00
V2S		TOTAL			157	165	95.15
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	15.340s	297.874us	44	50	88.00
V3		TOTAL			44	50	88.00
		TOTAL			1078	1110	97.12

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	5	71.43
V2	16	16	11	68.75
V2S	6	6	4	66.67
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.82	99.10	98.03	98.46	100.00	99.11	98.41	91.66

Failure Buckets

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 18 failures:

Test keymgr_tl_intg_err has 4 failures.

1.keymgr_tl_intg_err.602092386
Line 258, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[16] & 'hffffffff)))'
  UVM_ERROR @   9866178 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] attest_sw_binding_3_rd_A
  UVM_INFO @   9866178 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

7.keymgr_tl_intg_err.294534869
Line 276, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[20] & 'hffffffff)))'
  UVM_ERROR @  75245099 ps: (keymgr_csr_assert_fpv.sv:441) [ASSERT FAILED] attest_sw_binding_7_rd_A
  UVM_INFO @  75245099 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_csr_bit_bash has 4 failures.

1.keymgr_csr_bit_bash.3707158805
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @ 295716802 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @ 295716802 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

2.keymgr_csr_bit_bash.2733835644
Line 254, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[16] & 'hffffffff)))'
  UVM_ERROR @ 479314349 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] attest_sw_binding_3_rd_A
  UVM_INFO @ 479314349 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_shadow_reg_errors_with_csr_rw has 4 failures.

2.keymgr_shadow_reg_errors_with_csr_rw.2997779049
Line 258, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @  71764887 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @  71764887 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

4.keymgr_shadow_reg_errors_with_csr_rw.2735472927
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[19] & 'hffffffff)))'
  UVM_ERROR @  20540221 ps: (keymgr_csr_assert_fpv.sv:436) [ASSERT FAILED] attest_sw_binding_6_rd_A
  UVM_INFO @  20540221 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_same_csr_outstanding has 2 failures.

2.keymgr_same_csr_outstanding.2410106096
Line 256, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[16] & 'hffffffff)))'
  UVM_ERROR @  88322227 ps: (keymgr_csr_assert_fpv.sv:421) [ASSERT FAILED] attest_sw_binding_3_rd_A
  UVM_INFO @  88322227 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

16.keymgr_same_csr_outstanding.2567063320
Line 254, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[12] & 'hffffffff)))'
  UVM_ERROR @  76266037 ps: (keymgr_csr_assert_fpv.sv:401) [ASSERT FAILED] sealing_sw_binding_7_rd_A
  UVM_INFO @  76266037 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_csr_rw has 4 failures.

7.keymgr_csr_rw.3369280965
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @   3773020 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @   3773020 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

10.keymgr_csr_rw.2508645945
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @  25322037 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @  25322037 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 4 failures:
- Test keymgr_stress_all_with_rand_reset has 4 failures.
  - 27.keymgr_stress_all_with_rand_reset.1228103497
    Line 914, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/27.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 139203422 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 139203422 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 32.keymgr_stress_all_with_rand_reset.1597011710
    Line 1204, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/32.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 655619349 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 655619349 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 2 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1019) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 2 failures:

Test keymgr_kmac_rsp_err has 2 failures.

18.keymgr_kmac_rsp_err.1879593702
Line 509, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/18.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  46411023 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (28931355155510039809335513957072230219123189615693029102291992239391099081702028585551120987277780082453252808081644836663868423574052186186892047415486135256877494451371376876517985674444756995590998754996322307118459360640066614587415752432399052343673801352547784723621288235098055148503065196072705703933788837193770277047925068685945239934781639958753752199863516636272113279665692734008559633036364145498344235430361791121393780 [0xf373f27d3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f98029dd13c4c3068b27bd2a6c1994ed421d6f1d49c5e8495f4d49c7ae18d589cda9db37083c641b3848a527c9997e8cf897bf64e5b1f8c3e570df9ba0e1682a245998aa3e5d3f42658ba3ff58cadbae6805f009e0302287e33d8973312125b7164e8501568a543a13ffcb4368b9026fa3d092c76c338601207e2eb709e916971bbf85f5fa2fa346ba728e9be03e7d0874] vs 28931355155510039809335513957072230219123189615693029102291992239391099081702028585551120987277780082453252808081644836663868423574052186186892047415486135256877494451371376876517985674444756995590998754996322307118459360640066614587415752432399052343673801352547784723621288235098055148503065196072705703933788837193770277047925068685945239934781639958753752199863516636272113279665692734008559633036364145498344235430361791121393780 [0xf373f27d3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f98029dd13c4c3068b27bd2a6c1994ed421d6f1d49c5e8495f4d49c7ae18d589cda9db37083c641b3848a527c9997e8cf897bf64e5b1f8c3e570df9ba0e1682a245998aa3e5d3f42658ba3ff58cadbae6805f009e0302287e33d8973312125b7164e8501568a543a13ffcb4368b9026fa3d092c76c338601207e2eb709e916971bbf85f5fa2fa346ba728e9be03e7d0874]) cdi_type: Attestation
  
   DiversificationKey act: 0xd092c76c338601207e2eb709e916971bbf85f5fa2fa346ba728e9be03e7d0874, exp: 0xd092c76c338601207e2eb709e916971bbf85f5fa2fa346ba728e9be03e7d0874
   RomDigests act: 0x97bf64e5b1f8c3e570df9ba0e1682a245998aa3e5d3f42658ba3ff58cadbae6805f009e0302287e33d8973312125b7164e8501568a543a13ffcb4368b9026fa3, exp: 0x97bf64e5b1f8c3e570df9ba0e1682a245998aa3e5d3f42658ba3ff58cadbae6805f009e0302287e33d8973312125b7164e8501568a543a13ffcb4368b9026fa3
   HealthMeasurement act: 0xa9db37083c641b3848a527c9997e8cf8, exp: 0xa9db37083c641b3848a527c9997e8cf8

48.keymgr_kmac_rsp_err.387647434
Line 403, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/48.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 529457552 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (396115856434943208271198492316563781685478572028828858652382899111688069690597462858601800772371288097565267338454348013429843691923149236507015072874187380909323064476438776898636705030707488320136468719152975605004140742824125775614470793530196114694642567669729413354168540607157080508951260421195604223088961937352754067331061035817977220811391810815607049858581435445795821787838795403035265081465743347024617649545949524999564333461300739164805366426680840088296011266340999337824618920765105099 [0x7ba3242500000000ef5bf90600000000c93811fce073ca3f00000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9301339ea24ca9736689622642338e7f578ce14fd80c3cb115f7309192596e544ba6949cc342615e1ef71c74accc2cce9049a967f915311ca117fb497d63a4fe0692b566ea5ebcb46b20ffd000c19a7cc867fe89f700e523eb5d09fc640b47a2a3ae2c72537400d8ccf903e915733f7885665822c7e738514ae008e2ee4775944b3627ce5589d2623a3888d34d84923cb] vs 396115856434943208271198492316563781685478572028828858652382899111688069690597462858601800772371288097565267338454348013429843691923149236507015072874187380909323064476438776898636705030707488320136468719152975605004140742824125775614470793530196114694642567669729413354168540607157080508951260421195604223088961937352754067331061035817977220811391810815607049858581435445795821787838795403035265081465743347024617649545949524999564333461300739164805366426680840088296011266340999337824618920765105099 [0x7ba3242500000000ef5bf90600000000c93811fce073ca3f00000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9301339ea24ca9736689622642338e7f578ce14fd80c3cb115f7309192596e544ba6949cc342615e1ef71c74accc2cce9049a967f915311ca117fb497d63a4fe0692b566ea5ebcb46b20ffd000c19a7cc867fe89f700e523eb5d09fc640b47a2a3ae2c72537400d8ccf903e915733f7885665822c7e738514ae008e2ee4775944b3627ce5589d2623a3888d34d84923cb]) cdi_type: Attestation
  
   DiversificationKey act: 0x5665822c7e738514ae008e2ee4775944b3627ce5589d2623a3888d34d84923cb, exp: 0x5665822c7e738514ae008e2ee4775944b3627ce5589d2623a3888d34d84923cb
   RomDigests act: 0x49a967f915311ca117fb497d63a4fe0692b566ea5ebcb46b20ffd000c19a7cc867fe89f700e523eb5d09fc640b47a2a3ae2c72537400d8ccf903e915733f788, exp: 0x49a967f915311ca117fb497d63a4fe0692b566ea5ebcb46b20ffd000c19a7cc867fe89f700e523eb5d09fc640b47a2a3ae2c72537400d8ccf903e915733f788
   HealthMeasurement act: 0xba6949cc342615e1ef71c74accc2cce9, exp: 0xba6949cc342615e1ef71c74accc2cce9

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 2 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 19.keymgr_stress_all_with_rand_reset.1596889049
    Line 1142, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/19.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 250229289 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 250229289 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_hwsw_invalid_input has 1 failures.
  - 37.keymgr_hwsw_invalid_input.1069699980
    Line 474, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/37.keymgr_hwsw_invalid_input/latest/run.log
```
  UVM_ERROR @  81003570 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  81003570 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 2 failures:

Test keymgr_stress_all has 2 failures.

34.keymgr_stress_all.2146770442
Line 1763, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/34.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 693509028 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (3204220478 [0xbefc863e] vs 2067683335 [0x7b3e5807]) reg name: keymgr_reg_block.sw_share1_output_1
  UVM_INFO @ 693509028 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

36.keymgr_stress_all.3625765444
Line 1697, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 213396710 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (111964292 [0x6ac7084] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_7
  UVM_INFO @ 213396710 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_scoreboard.sv:241) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 3.keymgr_sync_async_fault_cross.3957662310
    Line 313, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 323755530 ps: (cip_base_scoreboard.sv:241) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 323755530 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:633) [scoreboard] Check failed act_state == addr_phase_working_state (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 10.keymgr_stress_all.236608721
    Line 1861, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 1542211444 ps: (keymgr_scoreboard.sv:633) [uvm_test_top.env.scoreboard] Check failed act_state == addr_phase_working_state (1 [0x1] vs 6 [0x6])
  UVM_INFO @ 1542211444 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

UVM_ERROR (keymgr_scoreboard.sv:1017) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 1 failures:

Test keymgr_stress_all_with_rand_reset has 1 failures.

22.keymgr_stress_all_with_rand_reset.2011692600
Line 1086, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/22.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 696975260 ps: (keymgr_scoreboard.sv:1017) [uvm_test_top.env.scoreboard] Check failed act == exp (182463715370630723710968994570116925559818010519459598781963429969185998874110246940780553795080604506983879525957142963449793514282058457727110203393339340795920156814193031364342281491359305456484073261460223444371976378620050124597362948126059601468314019673337931571260117838914071634268106023076221718150286279409810483937356356741044979869838690458764702772106391092193679441960285716772397943444086949286042647630351223333629689060373512640856630035614074839720032582025137451694911972307757398 [0x38f388d8ad9b1a0f88ab89cd53e09795fbd455199c5edaba8d001b8ec69e890d3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9e328e3b858f2cfc6e935fc04e28bc1071faf64f520f8ad36b28a2aefa135efa8c5ebe78c9ba5d040d581b08476d95d4286bb971f62b475acdd1babd4b66920e285f81bb19d64c16593b0dccc954a3dd6b5e5860897d7a0ed693d4ec76bf4290818a4f3d61fa9580c99a7803067368295d9800604fddee8fda6cf07df33233bdade53e5246b7fb3bccb440da9fd5de956] vs 781873943537550110538348113441966407068341039375715496420917234313000964310145477466802537142484410298046911482114745579500128847683054413829077043504153072812817253256379301862440892064637964139706191914466726510793262509375250925105443321541025274486423271701074260281995888025334600369907072940423861399120154090033203806777183453282492254424090413018014773344670233157223255009439088226721104184326681084550737492236745857394270418185614608402378347296762932737229427064907309570452224680344283478 [0xf40aa8657c8a4b7b0db8365a81e8910d4e226d3c53359b61bc05d515eed1a70b3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9e328e3b858f2cfc6e935fc04e28bc1071faf64f520f8ad36b28a2aefa135efa8c5ebe78c9ba5d040d581b08476d95d4286bb971f62b475acdd1babd4b66920e285f81bb19d64c16593b0dccc954a3dd6b5e5860897d7a0ed693d4ec76bf4290818a4f3d61fa9580c99a7803067368295d9800604fddee8fda6cf07df33233bdade53e5246b7fb3bccb440da9fd5de956]) cdi_type: Attestation
  
   DiversificationKey act: 0xd9800604fddee8fda6cf07df33233bdade53e5246b7fb3bccb440da9fd5de956, exp: 0xd9800604fddee8fda6cf07df33233bdade53e5246b7fb3bccb440da9fd5de956
   RomDigests act: 0x86bb971f62b475acdd1babd4b66920e285f81bb19d64c16593b0dccc954a3dd6b5e5860897d7a0ed693d4ec76bf4290818a4f3d61fa9580c99a7803067368295, exp: 0x86bb971f62b475acdd1babd4b66920e285f81bb19d64c16593b0dccc954a3dd6b5e5860897d7a0ed693d4ec76bf4290818a4f3d61fa9580c99a7803067368295
   HealthMeasurement act: 0xc5ebe78c9ba5d040d581b08476d95d42, exp: 0xc5ebe78c9ba5d040d581b08476d95d42

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share0_output_*` has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 40.keymgr_stress_all.1155704470
    Line 948, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/40.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 228319279 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (709714889 [0x2a4d63c9] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share0_output_3
  UVM_INFO @ 228319279 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```