KEYMGR Simulation Results

Thursday February 08 2024 20:05:57 UTC

GitHub Revision: `5470b05185`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 2313004684

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	56.610s	2.006ms	50	50	100.00
V1	random	keymgr_random	37.310s	1.086ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.560s	44.374us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.430s	51.521us	17	20	85.00
V1	csr_bit_bash	keymgr_csr_bit_bash	24.500s	879.099us	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	14.250s	754.818us	4	5	80.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.840s	73.020us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.430s	51.521us	17	20	85.00
		keymgr_csr_aliasing	14.250s	754.818us	4	5	80.00
V1		TOTAL			151	155	97.42
V2	cfgen_during_op	keymgr_cfg_regwen	1.687m	2.167ms	49	50	98.00
V2	sideload	keymgr_sideload	1.063m	21.120ms	50	50	100.00
		keymgr_sideload_kmac	53.500s	7.579ms	50	50	100.00
		keymgr_sideload_aes	54.090s	4.110ms	50	50	100.00
		keymgr_sideload_otbn	1.009m	2.263ms	49	50	98.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	27.570s	6.600ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	11.560s	500.262us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.173m	4.560ms	47	50	94.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.456m	32.891ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.243m	10.158ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	24.220s	1.132ms	50	50	100.00
V2	stress_all	keymgr_stress_all	8.139m	19.965ms	49	50	98.00
V2	intr_test	keymgr_intr_test	1.050s	22.165us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.020s	34.925us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	6.210s	1.824ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	6.210s	1.824ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.560s	44.374us	5	5	100.00
		keymgr_csr_rw	1.430s	51.521us	17	20	85.00
		keymgr_csr_aliasing	14.250s	754.818us	4	5	80.00
		keymgr_same_csr_outstanding	3.530s	341.632us	16	20	80.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.560s	44.374us	5	5	100.00
		keymgr_csr_rw	1.430s	51.521us	17	20	85.00
		keymgr_csr_aliasing	14.250s	754.818us	4	5	80.00
		keymgr_same_csr_outstanding	3.530s	341.632us	16	20	80.00
V2		TOTAL			730	740	98.65
V2S	sec_cm_additional_check	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
		keymgr_tl_intg_err	33.970s	5.737ms	16	20	80.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	36.380s	2.440ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	36.380s	2.440ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	36.380s	2.440ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	36.380s	2.440ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	15.220s	446.821us	12	20	60.00
V2S	prim_count_check	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	33.970s	5.737ms	16	20	80.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	36.380s	2.440ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.687m	2.167ms	49	50	98.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	37.310s	1.086ms	50	50	100.00
		keymgr_csr_rw	1.430s	51.521us	17	20	85.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	37.310s	1.086ms	50	50	100.00
		keymgr_csr_rw	1.430s	51.521us	17	20	85.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	37.310s	1.086ms	50	50	100.00
		keymgr_csr_rw	1.430s	51.521us	17	20	85.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	11.560s	500.262us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.243m	10.158ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.243m	10.158ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	37.310s	1.086ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	55.130s	5.928ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	55.010s	1.836ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	11.560s	500.262us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	55.010s	1.836ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	55.010s	1.836ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	55.010s	1.836ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	1.813m	4.601ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	55.010s	1.836ms	49	50	98.00
V2S		TOTAL			152	165	92.12
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	13.970s	187.996us	46	50	92.00
V3		TOTAL			46	50	92.00
		TOTAL			1079	1110	97.21

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	5	71.43
V2	16	16	11	68.75
V2S	6	6	3	50.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.80	99.10	97.95	98.37	100.00	99.11	98.41	91.63

Failure Buckets

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 20 failures:

Test keymgr_tl_intg_err has 4 failures.

0.keymgr_tl_intg_err.203333757
Line 258, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @  10138233 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @  10138233 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

1.keymgr_tl_intg_err.2660535654
Line 261, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[6] & 'hffffffff)))'
  UVM_ERROR @  34942738 ps: (keymgr_csr_assert_fpv.sv:371) [ASSERT FAILED] sealing_sw_binding_1_rd_A
  UVM_INFO @  34942738 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_shadow_reg_errors_with_csr_rw has 8 failures.

1.keymgr_shadow_reg_errors_with_csr_rw.2634852741
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[15] & 'hffffffff)))'
  UVM_ERROR @   9260474 ps: (keymgr_csr_assert_fpv.sv:416) [ASSERT FAILED] attest_sw_binding_2_rd_A
  UVM_INFO @   9260474 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

2.keymgr_shadow_reg_errors_with_csr_rw.2100275745
Line 261, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[20] & 'hffffffff)))'
  UVM_ERROR @  43179627 ps: (keymgr_csr_assert_fpv.sv:441) [ASSERT FAILED] attest_sw_binding_7_rd_A
  UVM_INFO @  43179627 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 6 more failures.

Test keymgr_csr_aliasing has 1 failures.

1.keymgr_csr_aliasing.737244676
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_aliasing/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @ 497251519 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @ 497251519 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_same_csr_outstanding has 4 failures.

4.keymgr_same_csr_outstanding.1649456733
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[14] & 'hffffffff)))'
  UVM_ERROR @   8013447 ps: (keymgr_csr_assert_fpv.sv:411) [ASSERT FAILED] attest_sw_binding_1_rd_A
  UVM_INFO @   8013447 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

11.keymgr_same_csr_outstanding.4139391408
Line 254, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/11.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[6] & 'hffffffff)))'
  UVM_ERROR @ 287845892 ps: (keymgr_csr_assert_fpv.sv:371) [ASSERT FAILED] sealing_sw_binding_1_rd_A
  UVM_INFO @ 287845892 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Test keymgr_csr_rw has 3 failures.

8.keymgr_csr_rw.24587864
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/8.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[15] & 'hffffffff)))'
  UVM_ERROR @  35963485 ps: (keymgr_csr_assert_fpv.sv:416) [ASSERT FAILED] attest_sw_binding_2_rd_A
  UVM_INFO @  35963485 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

11.keymgr_csr_rw.1023296457
Line 256, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/11.keymgr_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @  84168638 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @  84168638 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 4 failures:

Test keymgr_custom_cm has 1 failures.

3.keymgr_custom_cm.3029162562
Line 401, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_custom_cm/latest/run.log

  UVM_ERROR @  20724210 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  20724210 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sideload_otbn has 1 failures.

10.keymgr_sideload_otbn.3756926575
Line 276, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_sideload_otbn/latest/run.log

  UVM_ERROR @  39647883 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  39647883 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 1 failures.

20.keymgr_stress_all.2281914941
Line 502, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/20.keymgr_stress_all/latest/run.log

  UVM_ERROR @  94284446 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:2000
  UVM_INFO @  94284446 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_cfg_regwen has 1 failures.

29.keymgr_cfg_regwen.171890759
Line 347, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/29.keymgr_cfg_regwen/latest/run.log

  UVM_ERROR @  51849119 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  51849119 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1019) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 3 failures:

Test keymgr_kmac_rsp_err has 3 failures.

0.keymgr_kmac_rsp_err.1776103243
Line 387, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  28902287 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (353465693721288060275540349183069193872661332605407361488470814579696376778279422103980286322079182823134256582020355670474038866727427617180454879295292379533806969968123191729423706863037244157146455527549647074442855570689925808640342316115614245910552801619310381774372589371625033697097205709125927530914759315122473259907107830488923960934418007151677080619088347158526012323920885312080484364546626118093567205823372355217658829797746860600009274921687078354206095716652322346322394893862431608 [0x6e533bbe993d9e4715292225b62a611f99bfe3740abff1c32d12a1405c1936653a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 353465693721288060275540349183069193872661332605407361488470814579696376778279422103980286322079182823134256582020355670474038866727427617180454879295292379533806969968123191729423706863037244157146455527549647074442855570689925808640342316115614245910552801619310381774372589371625033697097205709125927530914759315122473259907107830488923960934418007151677080619088347158526012323920885312080484364546626118093567205823372355217658829797746860600009274921687078354206095716652322346322394893862431608 [0x6e533bbe993d9e4715292225b62a611f99bfe3740abff1c32d12a1405c1936653a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigests act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

27.keymgr_kmac_rsp_err.403435127
Line 308, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/27.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  12585169 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (752306690132897166819010774820622588310728193967368383198541505274242058060766616038320875109076944468284436070653075404952971707655537881327176324510945915167029066143653696307630981129368109479353730924826285665536628280977081895979659060565913602517752932824564897531062887555560218832248393284160490565629478124965059349605766766254560396753311709659227554997880406844611461686755977802619098883257216917451204746130519502209695595985402148157614743312869742040957647935391053082721085000846535544 [0xead01f9b34034cae0000000000000000631b151142c7273b00000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 752306690132897166819010774820622588310728193967368383198541505274242058060766616038320875109076944468284436070653075404952971707655537881327176324510945915167029066143653696307630981129368109479353730924826285665536628280977081895979659060565913602517752932824564897531062887555560218832248393284160490565629478124965059349605766766254560396753311709659227554997880406844611461686755977802619098883257216917451204746130519502209695595985402148157614743312869742040957647935391053082721085000846535544 [0xead01f9b34034cae0000000000000000631b151142c7273b00000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigests act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507a20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

... and 1 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1017) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

20.keymgr_stress_all_with_rand_reset.1296506733
Line 606, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/20.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 503049372 ps: (keymgr_scoreboard.sv:1017) [uvm_test_top.env.scoreboard] Check failed act == exp (566486302379191644299570539753568678526790682472318291301309121279248630681679834430274831949552581701815043849482782702587479406670137390400748158422852189203523472161960879534697165022938585118029281312021148744939094756026052566061583429745790695070377844742024614251177514965357702494964918202583390999836301177969179643556119549881402455289787878317529852860294809842453976899011002241944865687445500381790367232567322518185488266127512658379994490093850741167069231976875288286755006224308780545 [0xb0d060d2068c5aa0e34c1eae80323b8139d4f50b5bfb3339a082db791f8714fa3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f96873d78356c0cc30c770dcc03fde3d8b188d8d3cf9b80bd2fa4eda7ff57cd0553137d777dc1c99211225d4558732d532250430340d89aaf72cdc63fcc17596eca94e386713cf25008b3646b8a52a733abd36f939d4e81a4c553f043a21fe30e5ceefe65091d907f6000773a2dd7f8cb451e70e11dee877773dfb02ae9d6aac0daaaddf7340ae16824a76c9e42bda3e01] vs 107471505595052665197131067172373607287444128924273622793983000770806815157221198181600141185998713315333026777864579777604545927119022856116237239466314257911847222083944705190073794462916211116895792979712775135007341711335026652667530288827941166375081267182474722717597830426089048732954107773981684155289080140598997883461558168860738307096536721912461849544538949784726199022436392881488546429578861175039276819601826912346948495788322457199245346357728057136640798131070966933148636537975094785 [0x218b60415127691ae5d16de3e0daf82da139c28bb90ad9b28803087c324c39ca3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f96873d78356c0cc30c770dcc03fde3d8b188d8d3cf9b80bd2fa4eda7ff57cd0553137d777dc1c99211225d4558732d532250430340d89aaf72cdc63fcc17596eca94e386713cf25008b3646b8a52a733abd36f939d4e81a4c553f043a21fe30e5ceefe65091d907f6000773a2dd7f8cb451e70e11dee877773dfb02ae9d6aac0daaaddf7340ae16824a76c9e42bda3e01]) cdi_type: Attestation
  
   DiversificationKey act: 0x51e70e11dee877773dfb02ae9d6aac0daaaddf7340ae16824a76c9e42bda3e01, exp: 0x51e70e11dee877773dfb02ae9d6aac0daaaddf7340ae16824a76c9e42bda3e01
   RomDigests act: 0x250430340d89aaf72cdc63fcc17596eca94e386713cf25008b3646b8a52a733abd36f939d4e81a4c553f043a21fe30e5ceefe65091d907f6000773a2dd7f8cb4, exp: 0x250430340d89aaf72cdc63fcc17596eca94e386713cf25008b3646b8a52a733abd36f939d4e81a4c553f043a21fe30e5ceefe65091d907f6000773a2dd7f8cb4
   HealthMeasurement act: 0x3137d777dc1c99211225d4558732d532, exp: 0x3137d777dc1c99211225d4558732d532

28.keymgr_stress_all_with_rand_reset.1046162447
Line 1478, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/28.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 835512290 ps: (keymgr_scoreboard.sv:1017) [uvm_test_top.env.scoreboard] Check failed act == exp (1605930754565491250042339328357387143542526556207941497641627338815315533913318469771375836070482013604331089363653418745646618914612066774505269166689868130819972615034040417635941635972855869620346628258723220690330360582168857747337178326515866529392939875371338333340908534102012003015657560731659352559385020669845958090365953656816900104235601975571585846821933593971248153271434182801072751942352701700988610059047532 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f91684095976524208c3118962f446fa8c09197ba9b866310df2a1c637a27223a02ae76fc3085982e843eeaa5c5ade7acb860ab6dfd5cee0c1987b1b95d8db7a6ab5863378ac367642095711e264e082fa00d1c1cfa45cc2534e9e0af0cfda4698bc2627e7281425ef241b824ef714e8d2eb73a3e287a435a425f21920a08c1ead5e916cbdd0cef0affff6630fc4fe126c] vs 64435504677361263684944254684226284933560921624157267824354311989473741750154901185359394183638462901796412087210951612311991713865223382029796503316905667556738670501965787203967083914193544923342466275043930759480462197594718698203073608490820878190482497909455732856702573117076486828728494489513642289976519211745720328221342772319160554171347709358473590106896328056998400799613911617838366539821958666808351821684092601278432459269267418959975020 [0x1d64be460000000058782bcb3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f91684095976524208c3118962f446fa8c09197ba9b866310df2a1c637a27223a02ae76fc3085982e843eeaa5c5ade7acb860ab6dfd5cee0c1987b1b95d8db7a6ab5863378ac367642095711e264e082fa00d1c1cfa45cc2534e9e0af0cfda4698bc2627e7281425ef241b824ef714e8d2eb73a3e287a435a425f21920a08c1ead5e916cbdd0cef0affff6630fc4fe126c]) cdi_type: Attestation
  
   DiversificationKey act: 0xeb73a3e287a435a425f21920a08c1ead5e916cbdd0cef0affff6630fc4fe126c, exp: 0xeb73a3e287a435a425f21920a08c1ead5e916cbdd0cef0affff6630fc4fe126c
   RomDigests act: 0x860ab6dfd5cee0c1987b1b95d8db7a6ab5863378ac367642095711e264e082fa00d1c1cfa45cc2534e9e0af0cfda4698bc2627e7281425ef241b824ef714e8d2, exp: 0x860ab6dfd5cee0c1987b1b95d8db7a6ab5863378ac367642095711e264e082fa00d1c1cfa45cc2534e9e0af0cfda4698bc2627e7281425ef241b824ef714e8d2
   HealthMeasurement act: 0x2ae76fc3085982e843eeaa5c5ade7acb, exp: 0x2ae76fc3085982e843eeaa5c5ade7acb

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 7.keymgr_stress_all_with_rand_reset.4180744971
    Line 687, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 181797547 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 181797547 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 47.keymgr_stress_all_with_rand_reset.3734748428
    Line 653, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/47.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 197091064 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2207910216 [0x839a0948] vs 2207910216 [0x839a0948])
  UVM_INFO @ 197091064 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```