KEYMGR Simulation Results

Thursday February 29 2024 20:02:24 UTC

GitHub Revision: `e0c4026501`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 37435771356197831901593787335841447308974032110687249165442170486932885997295

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.043m	6.756ms	50	50	100.00
V1	random	keymgr_random	1.610m	15.027ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.270s	23.616us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.530s	223.291us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	32.230s	4.408ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	13.380s	1.986ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.310s	119.976us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.530s	223.291us	20	20	100.00
		keymgr_csr_aliasing	13.380s	1.986ms	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	2.282m	5.079ms	50	50	100.00
V2	sideload	keymgr_sideload	1.189m	3.625ms	50	50	100.00
		keymgr_sideload_kmac	1.332m	7.673ms	50	50	100.00
		keymgr_sideload_aes	1.380m	8.049ms	50	50	100.00
		keymgr_sideload_otbn	58.190s	8.473ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	35.350s	2.650ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	24.610s	1.726ms	48	50	96.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.456m	7.722ms	44	50	88.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.940m	7.375ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.123m	7.312ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	15.210s	4.023ms	50	50	100.00
V2	stress_all	keymgr_stress_all	20.047m	129.059ms	49	50	98.00
V2	intr_test	keymgr_intr_test	0.950s	21.109us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.100s	111.462us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.140s	154.251us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.140s	154.251us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.270s	23.616us	5	5	100.00
		keymgr_csr_rw	1.530s	223.291us	20	20	100.00
		keymgr_csr_aliasing	13.380s	1.986ms	5	5	100.00
		keymgr_same_csr_outstanding	4.800s	121.414us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.270s	23.616us	5	5	100.00
		keymgr_csr_rw	1.530s	223.291us	20	20	100.00
		keymgr_csr_aliasing	13.380s	1.986ms	5	5	100.00
		keymgr_same_csr_outstanding	4.800s	121.414us	20	20	100.00
V2		TOTAL			731	740	98.78
V2S	sec_cm_additional_check	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
		keymgr_tl_intg_err	43.200s	5.980ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	11.420s	497.484us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	11.420s	497.484us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	11.420s	497.484us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	11.420s	497.484us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	13.680s	1.294ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	43.200s	5.980ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	11.420s	497.484us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.282m	5.079ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.610m	15.027ms	50	50	100.00
		keymgr_csr_rw	1.530s	223.291us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.610m	15.027ms	50	50	100.00
		keymgr_csr_rw	1.530s	223.291us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.610m	15.027ms	50	50	100.00
		keymgr_csr_rw	1.530s	223.291us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	24.610s	1.726ms	48	50	96.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.123m	7.312ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.123m	7.312ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.610m	15.027ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	22.400s	849.102us	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	16.470s	2.329ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	24.610s	1.726ms	48	50	96.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	16.470s	2.329ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	16.470s	2.329ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	16.470s	2.329ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	2.940m	31.228ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	16.470s	2.329ms	49	50	98.00
V2S		TOTAL			164	165	99.39
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	28.650s	3.871ms	22	50	44.00
V3		TOTAL			22	50	44.00
		TOTAL			1072	1110	96.58

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	13	81.25
V2S	6	6	5	83.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.83	99.07	98.10	98.53	100.00	99.11	98.41	91.58

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:788) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 20 failures:
- Test keymgr_stress_all_with_rand_reset has 20 failures.
  - 4.keymgr_stress_all_with_rand_reset.65826132834272819219471439666136747613951179750851287007916358020497504413407
    Line 321, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1697844072 ps: (cip_base_vseq.sv:788) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 1697844072 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 5.keymgr_stress_all_with_rand_reset.4088834148502742327817305367273803091505703895417353700885697613790812267682
    Line 410, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 2427051452 ps: (cip_base_vseq.sv:788) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 2427051452 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 18 more failures.
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 8 failures:
- Test keymgr_stress_all_with_rand_reset has 8 failures.
  - 11.keymgr_stress_all_with_rand_reset.47343610780116882438247997476093870491676802435819318435398511349007678171084
    Line 1133, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/11.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 353955827 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (3259365993 [0xc245fa69] vs 3259365993 [0xc245fa69])
  UVM_INFO @ 353955827 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 23.keymgr_stress_all_with_rand_reset.89789379851116607727381282459274475275192966662418261052322416213040370524176
    Line 1754, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/23.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 391930167 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (1932566814 [0x7330a11e] vs 1932566814 [0x7330a11e])
  UVM_INFO @ 391930167 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 6 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1013) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 6 failures:

Test keymgr_kmac_rsp_err has 6 failures.

3.keymgr_kmac_rsp_err.61075357944569247491401262587855411766343662160297833539635273488399446197256
Line 425, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  52646445 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (417592130649333697066673868070544024830618170865281644473901276803492306148448692465689563241820719543609671297376055240334934431550952613011292548968089791378661803463806742312565978110534967731631864077830021184661823225118305067282189396498329792079965396795431168692047831026275679191001238916597156524906573106880606374339458611706922251787216033787594209091482499156669882145576306976218448762071391676737127073515004 [0xf17a7af3d9ae760de2afd778a865b93e4c22e20a2484fea2b996a2b896005823a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ec1834490f2af510005d151c99cfb19446570a4ab5be5f43e30ef34c01aea420a800e19dace9b440f9f2dc5b826b1ddeefae77432efcd74cc4db4a6404947736d6df7690cbfc162b69e4b5d1cc133e1212c0cad8d6deb8b53626fc51ad1bee2b93c17bea12e6f96896cfb9bc03a505fc] vs 417592130649333697066673868070544024830618170865281644473901276803492306148448692465689563241820719543609671297376055240334934431550952613011292548968089791378661803463806742312565978110534967731631864077830021184661823225118305067282189396498329792079965396795431168692047831026275679191001238916597156524906573106880606374339458611706922251787216033787594209091482499156669882145576306976218448762071391676737127073515004 [0xf17a7af3d9ae760de2afd778a865b93e4c22e20a2484fea2b996a2b896005823a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ec1834490f2af510005d151c99cfb19446570a4ab5be5f43e30ef34c01aea420a800e19dace9b440f9f2dc5b826b1ddeefae77432efcd74cc4db4a6404947736d6df7690cbfc162b69e4b5d1cc133e1212c0cad8d6deb8b53626fc51ad1bee2b93c17bea12e6f96896cfb9bc03a505fc]) cdi_type: Attestation
  
   DiversificationKey act: 0x12c0cad8d6deb8b53626fc51ad1bee2b93c17bea12e6f96896cfb9bc03a505fc, exp: 0x12c0cad8d6deb8b53626fc51ad1bee2b93c17bea12e6f96896cfb9bc03a505fc
   RomDigest act: 0xefae77432efcd74cc4db4a6404947736d6df7690cbfc162b69e4b5d1cc133e12, exp: 0xefae77432efcd74cc4db4a6404947736d6df7690cbfc162b69e4b5d1cc133e12
   HealthMeasurement act: 0xa800e19dace9b440f9f2dc5b826b1dde, exp: 0xa800e19dace9b440f9f2dc5b826b1dde

25.keymgr_kmac_rsp_err.36453248737855027555592714867417612450485870994409536423374354810280523416929
Line 379, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/25.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  88686347 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (5047203790682263315733914920266310543311289565878890300910139625801780565247284861328469739088604936315832598678562664688381107777797871591161589278387561411709211991824686897965430748251637766943803160096156713183385751809783892168837917991375669048620030578035693050560011296183220092369204393259540910958552925570366158987926695344591122032317202503640840231638627088004112671908162113464413345939524264531458012090376959 [0xb669d95697af2dcf1bb3e118293a5f5a9d7b471eda993a93be03cc65005393b43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92a9e1efa45f0531bf3c242b1d4bf4a94d3bea63cad1378a248de525837dec6927c8209493885d53b90e341fdc0a021a24c63cf4dbafdac8a056383db8b878bc88b933328c5e6b3c40bed51b5f7de91919d86bf3b8634beab0e5a7cab2171c281b449762558ab668f1bb38e52c94ea2ff] vs 5047203790682263315733914920266310543311289565878890300910139625801780565247284861328469739088604936315832598678562664688381107777797871591161589278387561411709211991824686897965430748251637766943803160096156713183385751809783892168837917991375669048620030578035693050560011296183220092369204393259540910958552925570366158987926695344591122032317202503640840231638627088004112671908162113464413345939524264531458012090376959 [0xb669d95697af2dcf1bb3e118293a5f5a9d7b471eda993a93be03cc65005393b43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92a9e1efa45f0531bf3c242b1d4bf4a94d3bea63cad1378a248de525837dec6927c8209493885d53b90e341fdc0a021a24c63cf4dbafdac8a056383db8b878bc88b933328c5e6b3c40bed51b5f7de91919d86bf3b8634beab0e5a7cab2171c281b449762558ab668f1bb38e52c94ea2ff]) cdi_type: Attestation
  
   DiversificationKey act: 0x9d86bf3b8634beab0e5a7cab2171c281b449762558ab668f1bb38e52c94ea2ff, exp: 0x9d86bf3b8634beab0e5a7cab2171c281b449762558ab668f1bb38e52c94ea2ff
   RomDigest act: 0x4c63cf4dbafdac8a056383db8b878bc88b933328c5e6b3c40bed51b5f7de9191, exp: 0x4c63cf4dbafdac8a056383db8b878bc88b933328c5e6b3c40bed51b5f7de9191
   HealthMeasurement act: 0x7c8209493885d53b90e341fdc0a021a2, exp: 0x7c8209493885d53b90e341fdc0a021a2

... and 4 more failures.

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 2 failures:

Test keymgr_lc_disable has 1 failures.

9.keymgr_lc_disable.74220607860933970338887848562512077643390703829786643562055084979017225859440
Line 530, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/9.keymgr_lc_disable/latest/run.log

  UVM_ERROR @  40319346 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (2592946517 [0x9a8d3955] vs 2341475708 [0x8b90157c]) reg name: keymgr_reg_block.sw_share1_output_2
  UVM_INFO @  40319346 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 1 failures.

41.keymgr_stress_all.6522326436212349761062265740375310368351600928131431331682139495245253436633
Line 2566, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/41.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 400017606 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (1581074399 [0x5e3d47df] vs 2693391942 [0xa089e646]) reg name: keymgr_reg_block.sw_share1_output_7
  UVM_INFO @ 400017606 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:750) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share0_output_*` has 1 failures:
- Test keymgr_lc_disable has 1 failures.
  - 4.keymgr_lc_disable.107930145976720840915647336772303838906910970339779583369637923268815643107016
    Line 319, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_lc_disable/latest/run.log
```
  UVM_ERROR @  11256130 ps: (keymgr_scoreboard.sv:750) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (708206356 [0x2a365f14] vs 3742325184 [0xdf0f59c0]) reg name: keymgr_reg_block.sw_share0_output_2
  UVM_INFO @  11256130 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:287) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 1 failures:
- Test keymgr_custom_cm has 1 failures.
  - 10.keymgr_custom_cm.44064294800469864875263839339523538804717982925756836384875830063450006013729
    Line 338, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_custom_cm/latest/run.log
```
  UVM_ERROR @   7279339 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   7279339 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```