KEYMGR Simulation Results

Thursday March 14 2024 19:02:18 UTC

GitHub Revision: `e844018f2c`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 83239673812975098462159483702727474484560953854893181354811398969250076096082

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.179m	6.947ms	48	50	96.00
V1	random	keymgr_random	46.770s	8.380ms	49	50	98.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.150s	14.412us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.410s	24.224us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	32.930s	7.900ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	8.660s	128.846us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.620s	36.366us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.410s	24.224us	20	20	100.00
		keymgr_csr_aliasing	8.660s	128.846us	5	5	100.00
V1		TOTAL			152	155	98.06
V2	cfgen_during_op	keymgr_cfg_regwen	2.316m	2.712ms	50	50	100.00
V2	sideload	keymgr_sideload	46.020s	1.704ms	50	50	100.00
		keymgr_sideload_kmac	55.950s	7.789ms	50	50	100.00
		keymgr_sideload_aes	1.102m	19.246ms	50	50	100.00
		keymgr_sideload_otbn	42.130s	1.403ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	43.220s	1.768ms	49	50	98.00
V2	lc_disable	keymgr_lc_disable	28.500s	1.939ms	48	50	96.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.569m	7.334ms	44	50	88.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.210m	4.955ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.250m	2.325ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	18.720s	1.305ms	50	50	100.00
V2	stress_all	keymgr_stress_all	11.833m	76.201ms	48	50	96.00
V2	intr_test	keymgr_intr_test	0.890s	10.644us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.260s	28.752us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.680s	569.895us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.680s	569.895us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.150s	14.412us	5	5	100.00
		keymgr_csr_rw	1.410s	24.224us	20	20	100.00
		keymgr_csr_aliasing	8.660s	128.846us	5	5	100.00
		keymgr_same_csr_outstanding	4.200s	127.642us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.150s	14.412us	5	5	100.00
		keymgr_csr_rw	1.410s	24.224us	20	20	100.00
		keymgr_csr_aliasing	8.660s	128.846us	5	5	100.00
		keymgr_same_csr_outstanding	4.200s	127.642us	20	20	100.00
V2		TOTAL			729	740	98.51
V2S	sec_cm_additional_check	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
		keymgr_tl_intg_err	31.890s	12.356ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	25.570s	1.170ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	25.570s	1.170ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	25.570s	1.170ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	25.570s	1.170ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.010s	619.180us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	31.890s	12.356ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	25.570s	1.170ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.316m	2.712ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	46.770s	8.380ms	49	50	98.00
		keymgr_csr_rw	1.410s	24.224us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	46.770s	8.380ms	49	50	98.00
		keymgr_csr_rw	1.410s	24.224us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	46.770s	8.380ms	49	50	98.00
		keymgr_csr_rw	1.410s	24.224us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	28.500s	1.939ms	48	50	96.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.250m	2.325ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.250m	2.325ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	46.770s	8.380ms	49	50	98.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	33.130s	2.984ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	1.670m	6.891ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	28.500s	1.939ms	48	50	96.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	1.670m	6.891ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	1.670m	6.891ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	1.670m	6.891ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	1.120m	3.034ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	1.670m	6.891ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	22.750s	1.468ms	26	50	52.00
V3		TOTAL			26	50	52.00
		TOTAL			1072	1110	96.58

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	5	71.43
V2	16	16	12	75.00
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.88	99.07	98.03	99.07	100.00	99.11	98.41	91.51

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:827) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 23 failures:
- Test keymgr_stress_all_with_rand_reset has 23 failures.
  - 4.keymgr_stress_all_with_rand_reset.103853048604647662992149339854131164205953795499931943763704305622900586291383
    Line 884, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 199226133 ps: (cip_base_vseq.sv:827) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10002 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 199226133 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 5.keymgr_stress_all_with_rand_reset.106509306129655077153551777065058389009687651143826908224258423971620664771708
    Line 1041, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 3479187231 ps: (cip_base_vseq.sv:827) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 3479187231 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 21 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1013) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 6 failures:

Test keymgr_kmac_rsp_err has 6 failures.

24.keymgr_kmac_rsp_err.485552581662865205331286172355809055537468183949690818123518643653430883505
Line 454, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  43670868 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (999070477753328469065512477991692628860966342368725819121554906036393476454303557872978240705588787594102760700296426734731296165726099930171140443309858431409735576874909374094921154590670209780707985445350944965383849973620962734856156389410297875530683121238786274191672575728771207431873103662605196546198784862791945890087603254329790378947186894228459139020563738857149076986764366833508984724973841112694734 [0x9b150d7fcf144bda0000000000000000d127959ba04d431c61f478633a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f96881c83cf8fa343405ec763594f1b62ec1c0d0e59703e3564645de43ef06289712371739cdf61bbcfd342b0f7b114fccb2b08f9a343a4c829f8553bb76e18219b588482416a69c3049fd8d349b37abffd1d400a00abb6045b4a264a8a856f8985d43b8e824cfc7213b30a036cdd8ffce] vs 999070477753328469065512477991692628860966342368725819121554906036393476454303557872978240705588787594102760700296426734731296165726099930171140443309858431409735576874909374094921154590670209780707985445350944965383849973620962734856156389410297875530683121238786274191672575728771207431873103662605196546198784862791945890087603254329790378947186894228459139020563738857149076986764366833508984724973841112694734 [0x9b150d7fcf144bda0000000000000000d127959ba04d431c61f478633a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f96881c83cf8fa343405ec763594f1b62ec1c0d0e59703e3564645de43ef06289712371739cdf61bbcfd342b0f7b114fccb2b08f9a343a4c829f8553bb76e18219b588482416a69c3049fd8d349b37abffd1d400a00abb6045b4a264a8a856f8985d43b8e824cfc7213b30a036cdd8ffce]) cdi_type: Attestation
  
   DiversificationKey act: 0xd1d400a00abb6045b4a264a8a856f8985d43b8e824cfc7213b30a036cdd8ffce, exp: 0xd1d400a00abb6045b4a264a8a856f8985d43b8e824cfc7213b30a036cdd8ffce
   RomDigest act: 0xb2b08f9a343a4c829f8553bb76e18219b588482416a69c3049fd8d349b37abff, exp: 0xb2b08f9a343a4c829f8553bb76e18219b588482416a69c3049fd8d349b37abff
   HealthMeasurement act: 0x12371739cdf61bbcfd342b0f7b114fcc, exp: 0x12371739cdf61bbcfd342b0f7b114fcc

31.keymgr_kmac_rsp_err.72620354658921261263956741472652757756361141831092360875048323403221919230090
Line 281, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/31.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  29620323 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (13869088684237589750254652523805745455607483863801036890679746459828093165537632134051449499055074792047852673444112312967876431495625947140369589068609073747179788248723486094364639000707099094835223211210968664499551638165869403005481135336566590599912980500617315204490775108932623747023836566052044800475126239550772109303915781392197041900359 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f903bbeb959440c15644516fccae82c765968146fa1edf2079373fa26bd96182f944e2f26332954d06ecb4504338ebe609a4d5cb233786376197843b1115e397be20f91e108501e4c61f6d5cf5ee2c69831e4d21c2b189e31ab676236ade5a3df39ab147bcf014b9b65c84a736a5bdcb47] vs 13869088684237589750254652523805745455607483863801036890679746459828093165537632134051449499055074792047852673444112312967876431495625947140369589068609073747179788248723486094364639000707099094835223211210968664499551638165869403005481135336566590599912980500617315204490775108932623747023836566052044800475126239550772109303915781392197041900359 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f903bbeb959440c15644516fccae82c765968146fa1edf2079373fa26bd96182f944e2f26332954d06ecb4504338ebe609a4d5cb233786376197843b1115e397be20f91e108501e4c61f6d5cf5ee2c69831e4d21c2b189e31ab676236ade5a3df39ab147bcf014b9b65c84a736a5bdcb47]) cdi_type: Attestation
  
   DiversificationKey act: 0x1e4d21c2b189e31ab676236ade5a3df39ab147bcf014b9b65c84a736a5bdcb47, exp: 0x1e4d21c2b189e31ab676236ade5a3df39ab147bcf014b9b65c84a736a5bdcb47
   RomDigest act: 0xa4d5cb233786376197843b1115e397be20f91e108501e4c61f6d5cf5ee2c6983, exp: 0xa4d5cb233786376197843b1115e397be20f91e108501e4c61f6d5cf5ee2c6983
   HealthMeasurement act: 0x44e2f26332954d06ecb4504338ebe609, exp: 0x44e2f26332954d06ecb4504338ebe609

... and 4 more failures.

UVM_ERROR (cip_base_scoreboard.sv:287) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 5 failures:

Test keymgr_lc_disable has 1 failures.

9.keymgr_lc_disable.39442805261335359048439467057218711480985724885012325217275908776072845004087
Line 363, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/9.keymgr_lc_disable/latest/run.log

  UVM_ERROR @  22544911 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  22544911 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_random has 1 failures.

12.keymgr_random.110754230689784721514193544773677343860785699537199701869412084059777155481151
Line 414, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/12.keymgr_random/latest/run.log

  UVM_ERROR @  48696698 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  48696698 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_smoke has 2 failures.

14.keymgr_smoke.23841973533076281650984252948308754345500655256874924575268054335105923365411
Line 339, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_smoke/latest/run.log

  UVM_ERROR @ 502863866 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 502863866 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

16.keymgr_smoke.71748729618810255758467996097241800516287123377936021666081538736273247974974
Line 299, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_smoke/latest/run.log

  UVM_ERROR @   7329715 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   7329715 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_direct_to_disabled has 1 failures.

38.keymgr_direct_to_disabled.54823010505463802684141303936546385991000189153351460500340117880498996380854
Line 263, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/38.keymgr_direct_to_disabled/latest/run.log

  UVM_ERROR @   2455116 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   2455116 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 2 failures:
- Test keymgr_stress_all has 1 failures.
  - 14.keymgr_stress_all.108898165510768838758125380190625976455071151721428019807349501311058186041046
    Line 2820, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 1356039095 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (1668452486 [0x63729086] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_3
  UVM_INFO @ 1356039095 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_lc_disable has 1 failures.
  - 39.keymgr_lc_disable.102399801737902242405536300228486826531278580379964912467528090212631250576037
    Line 427, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/39.keymgr_lc_disable/latest/run.log
```
  UVM_ERROR @ 185504050 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (3876733348 [0xe71241a4] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_3
  UVM_INFO @ 185504050 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share0_output_*` has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 26.keymgr_stress_all.113930910568070865739678455681651365102600891966097652978839082882509254825544
    Line 2358, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/26.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 1972105424 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (2671664986 [0x9f3e5f5a] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share0_output_7
  UVM_INFO @ 1972105424 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 31.keymgr_stress_all_with_rand_reset.46054108154151317628357044916196754140715352763337762722703981149447011386922
    Line 1462, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/31.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 532587870 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (836504637 [0x31dc0c3d] vs 836504637 [0x31dc0c3d])
  UVM_INFO @ 532587870 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```