KEYMGR Simulation Results

Sunday May 21 2023 07:04:58 UTC

GitHub Revision: `e3fb01b5e`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 3002339765

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	28.430s	5.259ms	50	50	100.00
V1	random	keymgr_random	57.200s	2.492ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.550s	137.247us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.430s	30.258us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	30.410s	6.573ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	11.450s	3.518ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.080s	24.468us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.430s	30.258us	20	20	100.00
		keymgr_csr_aliasing	11.450s	3.518ms	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	1.430m	18.604ms	50	50	100.00
V2	sideload	keymgr_sideload	21.830s	2.228ms	49	50	98.00
		keymgr_sideload_kmac	1.144m	4.830ms	50	50	100.00
		keymgr_sideload_aes	51.120s	8.699ms	49	50	98.00
		keymgr_sideload_otbn	35.860s	5.771ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	18.410s	593.497us	50	50	100.00
V2	lc_disable	keymgr_lc_disable	37.100s	1.429ms	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.596m	34.665ms	46	50	92.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.360m	24.621ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	52.250s	1.934ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	17.740s	3.698ms	50	50	100.00
V2	stress_all	keymgr_stress_all	8.904m	102.296ms	48	50	96.00
V2	intr_test	keymgr_intr_test	1.030s	24.704us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.060s	24.536us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.080s	190.343us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.080s	190.343us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.550s	137.247us	5	5	100.00
		keymgr_csr_rw	1.430s	30.258us	20	20	100.00
		keymgr_csr_aliasing	11.450s	3.518ms	5	5	100.00
		keymgr_same_csr_outstanding	4.430s	127.789us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.550s	137.247us	5	5	100.00
		keymgr_csr_rw	1.430s	30.258us	20	20	100.00
		keymgr_csr_aliasing	11.450s	3.518ms	5	5	100.00
		keymgr_same_csr_outstanding	4.430s	127.789us	20	20	100.00
V2		TOTAL			732	740	98.92
V2S	sec_cm_additional_check	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
		keymgr_tl_intg_err	40.660s	4.018ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	31.820s	3.442ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	31.820s	3.442ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	31.820s	3.442ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	31.820s	3.442ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	15.720s	429.029us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	40.660s	4.018ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	31.820s	3.442ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.430m	18.604ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	57.200s	2.492ms	50	50	100.00
		keymgr_csr_rw	1.430s	30.258us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	57.200s	2.492ms	50	50	100.00
		keymgr_csr_rw	1.430s	30.258us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	57.200s	2.492ms	50	50	100.00
		keymgr_csr_rw	1.430s	30.258us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	37.100s	1.429ms	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	52.250s	1.934ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	52.250s	1.934ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	57.200s	2.492ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	26.830s	1.804ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	47.030s	9.564ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	37.100s	1.429ms	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	47.030s	9.564ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	47.030s	9.564ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	47.030s	9.564ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	5.361m	10.722ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	47.030s	9.564ms	49	50	98.00
V2S		TOTAL			164	165	99.39
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	16.310s	687.649us	45	50	90.00
V3		TOTAL			45	50	90.00
		TOTAL			1096	1110	98.74

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	12	75.00
V2S	6	6	5	83.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.85	99.09	98.17	98.51	100.00	99.08	98.38	91.71

Failure Buckets

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 5 failures:

Test keymgr_stress_all has 2 failures.

9.keymgr_stress_all.2571499105
Line 1195, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/9.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 510698267 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 510698267 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

11.keymgr_stress_all.4008344102
Line 2979, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/11.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 3865522419 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 3865522419 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_custom_cm has 1 failures.

12.keymgr_custom_cm.3671373050
Line 232, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/12.keymgr_custom_cm/latest/run.log

  UVM_ERROR @   8997259 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   8997259 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sideload_aes has 1 failures.

17.keymgr_sideload_aes.2603265956
Line 244, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/17.keymgr_sideload_aes/latest/run.log

  UVM_ERROR @  17198761 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  17198761 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sideload has 1 failures.

36.keymgr_sideload.3638021166
Line 265, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_sideload/latest/run.log

  UVM_ERROR @  17288689 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  17288689 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1012) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 4 failures:

Test keymgr_kmac_rsp_err has 4 failures.

8.keymgr_kmac_rsp_err.3556633714
Line 298, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/8.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  38956469 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (3428321046251009086248290732027722618671536853426117497905541813005902360687030770845890004879383012933886633325780819812445978164981088980880007193738003325452345980475387425959478476663011963203215066363713357101464984791663271378869258892962271822030958863786741004521670134787461626935708618225951785293766328640174454628396309281180038735748356448627759803397447422560248965872082195607734534511220818843231601076226171 [0x7be79635000000000000000076450ddc00000000284050f900000000a1a5047b3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92cfbf6a967e326c9daada2f5342d6aaa5b1ee7a1b9e3ebb2fa2a0ddf7cfe65337152f28f82b1964fbaf9b87175d5dcb0e424c59b5f57cbda2b65410ac098bd66450879d848e9abe47819e9bd9c2253e122423c178cc29e2d5950284e9a67195f38b64f69638d722f183a10ae6b90487b] vs 3428321046251009086248290732027722618671536853426117497905541813005902360687030770845890004879383012933886633325780819812445978164981088980880007193738003325452345980475387425959478476663011963203215066363713357101464984791663271378869258892962271822030958863786741004521670134787461626935708618225951785293766328640174454628396309281180038735748356448627759803397447422560248965872082195607734534511220818843231601076226171 [0x7be79635000000000000000076450ddc00000000284050f900000000a1a5047b3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92cfbf6a967e326c9daada2f5342d6aaa5b1ee7a1b9e3ebb2fa2a0ddf7cfe65337152f28f82b1964fbaf9b87175d5dcb0e424c59b5f57cbda2b65410ac098bd66450879d848e9abe47819e9bd9c2253e122423c178cc29e2d5950284e9a67195f38b64f69638d722f183a10ae6b90487b]) cdi_type: Attestation
  
   DiversificationKey act: 0x22423c178cc29e2d5950284e9a67195f38b64f69638d722f183a10ae6b90487b, exp: 0x22423c178cc29e2d5950284e9a67195f38b64f69638d722f183a10ae6b90487b
   RomDigest act: 0xe424c59b5f57cbda2b65410ac098bd66450879d848e9abe47819e9bd9c2253e1, exp: 0xe424c59b5f57cbda2b65410ac098bd66450879d848e9abe47819e9bd9c2253e1
   HealthMeasurement act: 0x7152f28f82b1964fbaf9b87175d5dcb0, exp: 0x7152f28f82b1964fbaf9b87175d5dcb0

16.keymgr_kmac_rsp_err.4068787292
Line 433, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  82074308 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (1835659947685905841842863247957925388733736364544183891046074240095418273500641847038269062769556765070961578416724065126507559388912955417162044472414732505768947599845187868141088867108348983886066378940029510737872087676027563600733898147087083416723452026791165457074456250573443630423667724162353191979191357626860507542856777044186422211938192867285289230083067769735691076080218539253626655213440884771284615978671139 [0x4257eed8fed7b854d461d7e250e89932abc37a147256cc8289fe5a34dc13e9f83a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92574583c19f8f5fe4a519e2d59a4b31a507a6bdb24d471ad914bc54a4ecec4d8d0844e56d6e0ef5d9ed8c57cc7c88c7434adfcfa6f50275117c7de4e352764e61d7c4685255bbd727bedf99de851a7946e38009c342a1609d7b788d97729cf7805830a6a57af5e50548864a94971dc23] vs 1835659947685905841842863247957925388733736364544183891046074240095418273500641847038269062769556765070961578416724065126507559388912955417162044472414732505768947599845187868141088867108348983886066378940029510737872087676027563600733898147087083416723452026791165457074456250573443630423667724162353191979191357626860507542856777044186422211938192867285289230083067769735691076080218539253626655213440884771284615978671139 [0x4257eed8fed7b854d461d7e250e89932abc37a147256cc8289fe5a34dc13e9f83a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f92574583c19f8f5fe4a519e2d59a4b31a507a6bdb24d471ad914bc54a4ecec4d8d0844e56d6e0ef5d9ed8c57cc7c88c7434adfcfa6f50275117c7de4e352764e61d7c4685255bbd727bedf99de851a7946e38009c342a1609d7b788d97729cf7805830a6a57af5e50548864a94971dc23]) cdi_type: Attestation
  
   DiversificationKey act: 0x6e38009c342a1609d7b788d97729cf7805830a6a57af5e50548864a94971dc23, exp: 0x6e38009c342a1609d7b788d97729cf7805830a6a57af5e50548864a94971dc23
   RomDigest act: 0x34adfcfa6f50275117c7de4e352764e61d7c4685255bbd727bedf99de851a794, exp: 0x34adfcfa6f50275117c7de4e352764e61d7c4685255bbd727bedf99de851a794
   HealthMeasurement act: 0xd0844e56d6e0ef5d9ed8c57cc7c88c74, exp: 0xd0844e56d6e0ef5d9ed8c57cc7c88c74

... and 2 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

7.keymgr_stress_all_with_rand_reset.1147419916
Line 458, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 333041659 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 333041659 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

18.keymgr_stress_all_with_rand_reset.3827452313
Line 461, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/18.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @  85300023 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @  85300023 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

26.keymgr_stress_all_with_rand_reset.1338881504
Line 1282, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/26.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 191108212 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (3691033813 [0xdc00b4d5] vs 3691033813 [0xdc00b4d5])
  UVM_INFO @ 191108212 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

41.keymgr_stress_all_with_rand_reset.2444973807
Line 438, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/41.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 351223407 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2933435522 [0xaed8ac82] vs 2933435522 [0xaed8ac82])
  UVM_INFO @ 351223407 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:671) [scoreboard] Check failed edn_fifos[*].used() == * (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 44.keymgr_stress_all_with_rand_reset.1172492674
    Line 504, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/44.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 137629670 ps: (keymgr_scoreboard.sv:671) [uvm_test_top.env.scoreboard] Check failed edn_fifos[0].used() == 2 (3 [0x3] vs 2 [0x2])
  UVM_INFO @ 137629670 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```