KEYMGR Simulation Results

Tuesday May 23 2023 07:02:27 UTC

GitHub Revision: `83db9403d`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 1254715506

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	40.110s	3.694ms	50	50	100.00
V1	random	keymgr_random	1.003m	2.650ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.480s	30.731us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.550s	105.094us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	15.830s	1.679ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	13.760s	649.917us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.980s	155.239us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.550s	105.094us	20	20	100.00
		keymgr_csr_aliasing	13.760s	649.917us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	1.551m	3.632ms	50	50	100.00
V2	sideload	keymgr_sideload	43.890s	6.853ms	50	50	100.00
		keymgr_sideload_kmac	1.224m	11.758ms	50	50	100.00
		keymgr_sideload_aes	1.023m	28.160ms	50	50	100.00
		keymgr_sideload_otbn	1.253m	7.767ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	25.950s	1.084ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	13.100s	1.724ms	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	40.060s	4.194ms	42	50	84.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.073m	9.821ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	48.860s	3.925ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	18.560s	2.153ms	50	50	100.00
V2	stress_all	keymgr_stress_all	16.834m	33.489ms	48	50	96.00
V2	intr_test	keymgr_intr_test	0.940s	20.182us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.010s	23.626us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	3.560s	2.191ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	3.560s	2.191ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.480s	30.731us	5	5	100.00
		keymgr_csr_rw	1.550s	105.094us	20	20	100.00
		keymgr_csr_aliasing	13.760s	649.917us	5	5	100.00
		keymgr_same_csr_outstanding	4.120s	125.654us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.480s	30.731us	5	5	100.00
		keymgr_csr_rw	1.550s	105.094us	20	20	100.00
		keymgr_csr_aliasing	13.760s	649.917us	5	5	100.00
		keymgr_same_csr_outstanding	4.120s	125.654us	20	20	100.00
V2		TOTAL			730	740	98.65
V2S	sec_cm_additional_check	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
		keymgr_tl_intg_err	40.190s	1.783ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	25.810s	2.849ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	25.810s	2.849ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	25.810s	2.849ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	25.810s	2.849ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	18.360s	566.641us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	40.190s	1.783ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	25.810s	2.849ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.551m	3.632ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.003m	2.650ms	50	50	100.00
		keymgr_csr_rw	1.550s	105.094us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.003m	2.650ms	50	50	100.00
		keymgr_csr_rw	1.550s	105.094us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.003m	2.650ms	50	50	100.00
		keymgr_csr_rw	1.550s	105.094us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	13.100s	1.724ms	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	48.860s	3.925ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	48.860s	3.925ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.003m	2.650ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	26.380s	10.690ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	59.730s	5.272ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	13.100s	1.724ms	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	59.730s	5.272ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	59.730s	5.272ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	59.730s	5.272ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	31.310s	9.159ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	59.730s	5.272ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	15.430s	242.855us	47	50	94.00
V3		TOTAL			47	50	94.00
		TOTAL			1097	1110	98.83

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	14	87.50
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.85	99.09	98.17	98.57	100.00	99.08	98.38	91.68

Failure Buckets

UVM_ERROR (keymgr_scoreboard.sv:1012) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 8 failures:

Test keymgr_kmac_rsp_err has 8 failures.

10.keymgr_kmac_rsp_err.107880096
Line 308, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 183278255 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (4101651858190805340951113480114562038150542003333602011839638878511703856160453532027306987998865471027121920379843049806451160099018279868905804416565202059139836321821655564530284442199016229199906022579186890737505673286995143237670119411076622231485530226080223349418375777215278492835151261059179487325017591859196102310468223839164144859760692214360712716552330666467479345719425798489169146297269858346389289727290232 [0x943d648c70026d3c000000003eed7e649f688ae8000000003e6a82ff000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 4101651858190805340951113480114562038150542003333602011839638878511703856160453532027306987998865471027121920379843049806451160099018279868905804416565202059139836321821655564530284442199016229199906022579186890737505673286995143237670119411076622231485530226080223349418375777215278492835151261059179487325017591859196102310468223839164144859760692214360712716552330666467479345719425798489169146297269858346389289727290232 [0x943d648c70026d3c000000003eed7e649f688ae8000000003e6a82ff000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigest act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

16.keymgr_kmac_rsp_err.546918856
Line 241, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 535133642 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (13869088684237589750254652523805745455607483863801036890679746459828093165538127981335114092620659895955123613355869342400016532540949205728335701513618985725133514101639194238534945649948618425891381024752800776438361079227059548818834780102961361243320496097767783154276089326609680551121139813053013287040383931096600290352413475538474834287302 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9f402c61d930c0fe1a4c08b1e592088080c88e5d3c2ecb8e37338daa553abb4a58c15387f5ca755bebffb7b86422da427b795705078a5affabfb75740abbc78b1fcf0dd0c80344d31d0c8ef6963fe9adfa696cbaf9206778fe4dea78a8a2a920d29a82a9768e3f71868655db677db5ac6] vs 13869088684237589750254652523805745455607483863801036890679746459828093165538127981335114092620659895955123613355869342400016532540949205728335701513618985725133514101639194238534945649948618425891381024752800776438361079227059548818834780102961361243320496097767783154276089326609680551121139813053013287040383931096600290352413475538474834287302 [0x3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9f402c61d930c0fe1a4c08b1e592088080c88e5d3c2ecb8e37338daa553abb4a58c15387f5ca755bebffb7b86422da427b795705078a5affabfb75740abbc78b1fcf0dd0c80344d31d0c8ef6963fe9adfa696cbaf9206778fe4dea78a8a2a920d29a82a9768e3f71868655db677db5ac6]) cdi_type: Attestation
  
   DiversificationKey act: 0xa696cbaf9206778fe4dea78a8a2a920d29a82a9768e3f71868655db677db5ac6, exp: 0xa696cbaf9206778fe4dea78a8a2a920d29a82a9768e3f71868655db677db5ac6
   RomDigest act: 0xb795705078a5affabfb75740abbc78b1fcf0dd0c80344d31d0c8ef6963fe9adf, exp: 0xb795705078a5affabfb75740abbc78b1fcf0dd0c80344d31d0c8ef6963fe9adf
   HealthMeasurement act: 0x8c15387f5ca755bebffb7b86422da427, exp: 0x8c15387f5ca755bebffb7b86422da427

... and 6 more failures.

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 2 failures:

Test keymgr_stress_all has 2 failures.

4.keymgr_stress_all.2659107046
Line 2734, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 52222143891 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 52222143891 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

18.keymgr_stress_all.3115875071
Line 897, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/18.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 324541914 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 324541914 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1010) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 1 failures:

Test keymgr_stress_all_with_rand_reset has 1 failures.

27.keymgr_stress_all_with_rand_reset.3497557929
Line 1259, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/27.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 119751968 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (5405447407871648586940328946932438573917551574948680941411627082445118664548429453550611545258541546419330606149832049282429091268574737629826418478713557894407504404771238601621390893029573743818088346726452995307712875666141694104871848682499806926145534848061601853675680264150448084079396944442188706470063516263484284641642577245781522286062604588957294629089846547802954447673690878932035731752697222686119910472263725 [0xc35c65fc8bcdb18c28a3696cd517dc7b867bb8f211520a572d7c8c0f3b66ad8a3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f99f8882464742d4aee015e0d917559b92bb8a84088301e9d79cc52e6ec34e87f7530fd3cbd0a0271d3e1f671089cefad97d122a7f96f7a6cfa8c36d691270574388b298effb4507c2b890de3cbee71e201d365dbce5ce2723194232fb2d729fce2ee0c227ab6e5816ffd0b734d35da82d] vs 5708199278073717684773556903141717561830386199288377790787108767956116435473595938001981597057725367083743545284061233960405435218079604681881072278316909809732104643550615718119227835128625369996949282687601817944623383768451368740021365526583499346874990370550555372245063884974249853836724949116701442421559779260756844248328137111972275455240066902042953212794876204758676855216675979620791332419661189523459942257633325 [0xce4d869d7adba465fb829f1cf6d45adc431c8bd8d6b5cd9c55fff11596990ccf3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f99f8882464742d4aee015e0d917559b92bb8a84088301e9d79cc52e6ec34e87f7530fd3cbd0a0271d3e1f671089cefad97d122a7f96f7a6cfa8c36d691270574388b298effb4507c2b890de3cbee71e201d365dbce5ce2723194232fb2d729fce2ee0c227ab6e5816ffd0b734d35da82d]) cdi_type: Attestation
  
   DiversificationKey act: 0x1d365dbce5ce2723194232fb2d729fce2ee0c227ab6e5816ffd0b734d35da82d, exp: 0x1d365dbce5ce2723194232fb2d729fce2ee0c227ab6e5816ffd0b734d35da82d
   RomDigest act: 0x7d122a7f96f7a6cfa8c36d691270574388b298effb4507c2b890de3cbee71e20, exp: 0x7d122a7f96f7a6cfa8c36d691270574388b298effb4507c2b890de3cbee71e20
   HealthMeasurement act: 0x530fd3cbd0a0271d3e1f671089cefad9, exp: 0x530fd3cbd0a0271d3e1f671089cefad9

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 40.keymgr_stress_all_with_rand_reset.3568427012
    Line 640, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/40.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @  65491642 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @  65491642 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 48.keymgr_stress_all_with_rand_reset.2060512887
    Line 997, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/48.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 252964839 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (1659741303 [0x62eda477] vs 1659741303 [0x62eda477])
  UVM_INFO @ 252964839 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```