KEYMGR Simulation Results

Monday May 22 2023 07:05:49 UTC

GitHub Revision: `e3fb01b5e`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 3641199223

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	47.900s	1.925ms	50	50	100.00
V1	random	keymgr_random	1.769m	13.813ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.380s	103.180us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.580s	57.142us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	15.760s	1.081ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	10.720s	504.510us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.920s	192.937us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.580s	57.142us	20	20	100.00
		keymgr_csr_aliasing	10.720s	504.510us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	1.290m	2.121ms	50	50	100.00
V2	sideload	keymgr_sideload	49.930s	5.222ms	50	50	100.00
		keymgr_sideload_kmac	1.064m	2.124ms	50	50	100.00
		keymgr_sideload_aes	1.402m	8.681ms	50	50	100.00
		keymgr_sideload_otbn	27.860s	1.520ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	35.780s	14.179ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	38.560s	1.407ms	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.232m	5.343ms	45	50	90.00
V2	invalid_sw_input	keymgr_sw_invalid_input	33.850s	5.446ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.311m	4.716ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	26.720s	1.043ms	49	50	98.00
V2	stress_all	keymgr_stress_all	10.117m	19.924ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.880s	60.184us	50	50	100.00
V2	alert_test	keymgr_alert_test	0.990s	16.554us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.480s	771.522us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.480s	771.522us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.380s	103.180us	5	5	100.00
		keymgr_csr_rw	1.580s	57.142us	20	20	100.00
		keymgr_csr_aliasing	10.720s	504.510us	5	5	100.00
		keymgr_same_csr_outstanding	4.100s	116.909us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.380s	103.180us	5	5	100.00
		keymgr_csr_rw	1.580s	57.142us	20	20	100.00
		keymgr_csr_aliasing	10.720s	504.510us	5	5	100.00
		keymgr_same_csr_outstanding	4.100s	116.909us	20	20	100.00
V2		TOTAL			734	740	99.19
V2S	sec_cm_additional_check	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
		keymgr_tl_intg_err	29.490s	16.871ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	19.760s	3.424ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	19.760s	3.424ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	19.760s	3.424ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	19.760s	3.424ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	15.570s	430.634us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	29.490s	16.871ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	19.760s	3.424ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.290m	2.121ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.769m	13.813ms	50	50	100.00
		keymgr_csr_rw	1.580s	57.142us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.769m	13.813ms	50	50	100.00
		keymgr_csr_rw	1.580s	57.142us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.769m	13.813ms	50	50	100.00
		keymgr_csr_rw	1.580s	57.142us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	38.560s	1.407ms	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.311m	4.716ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.311m	4.716ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.769m	13.813ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	25.530s	4.681ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	26.380s	2.759ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	38.560s	1.407ms	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	26.380s	2.759ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	26.380s	2.759ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	26.380s	2.759ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	33.390s	1.606ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	26.380s	2.759ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	15.380s	1.240ms	44	50	88.00
V3		TOTAL			44	50	88.00
		TOTAL			1098	1110	98.92

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	14	87.50
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.82	99.09	98.21	98.39	100.00	99.08	98.38	91.56

Failure Buckets

UVM_ERROR (keymgr_scoreboard.sv:1012) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 5 failures:

Test keymgr_kmac_rsp_err has 5 failures.

16.keymgr_kmac_rsp_err.598134876
Line 356, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 105226760 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (65527921283254052636599944564590437993367405523587036793168341596175605116292587197079686387320392389630246423997760858341358449861349059516134265634168259251935258213807719007919597888528837071074692068937404550320423291251556255210670005563447353618731499288873443498774687524475380930672984604073927573131504465149829726914860547166115246606635507389601177587489779191978661404702718681477902 [0xbba25c38000000000000000000000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9503cadd8a9d512b7da7a0924b69c772a4dfb35e70b1975e7146779daa80cc70e5ea8ec3490e5b4ae0fd3cbe9f48cd2ea505d8ab86fdba6ad3c79531d6449db58f47364ad12d130cdbf948acb9c1b1ec597b7adbc3bb847d18ca78750127450cd6e84dec1fe6fc60b3b8d1a598618f30e] vs 65527921283254052636599944564590437993367405523587036793168341596175605116292587197079686387320392389630246423997760858341358449861349059516134265634168259251935258213807719007919597888528837071074692068937404550320423291251556255210670005563447353618731499288873443498774687524475380930672984604073927573131504465149829726914860547166115246606635507389601177587489779191978661404702718681477902 [0xbba25c38000000000000000000000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9503cadd8a9d512b7da7a0924b69c772a4dfb35e70b1975e7146779daa80cc70e5ea8ec3490e5b4ae0fd3cbe9f48cd2ea505d8ab86fdba6ad3c79531d6449db58f47364ad12d130cdbf948acb9c1b1ec597b7adbc3bb847d18ca78750127450cd6e84dec1fe6fc60b3b8d1a598618f30e]) cdi_type: Attestation
  
   DiversificationKey act: 0x97b7adbc3bb847d18ca78750127450cd6e84dec1fe6fc60b3b8d1a598618f30e, exp: 0x97b7adbc3bb847d18ca78750127450cd6e84dec1fe6fc60b3b8d1a598618f30e
   RomDigest act: 0x505d8ab86fdba6ad3c79531d6449db58f47364ad12d130cdbf948acb9c1b1ec5, exp: 0x505d8ab86fdba6ad3c79531d6449db58f47364ad12d130cdbf948acb9c1b1ec5
   HealthMeasurement act: 0x5ea8ec3490e5b4ae0fd3cbe9f48cd2ea, exp: 0x5ea8ec3490e5b4ae0fd3cbe9f48cd2ea

17.keymgr_kmac_rsp_err.1525756816
Line 311, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/17.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  40145549 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (6707390009636963749785842126143380118397061316769764293488516074335641302924033377090610142487812533856185953148490380715366375464631372553300548003929694617441663059443348887627101886807512307111989153120101947324785234665280121502907129173354190530417267577934568114388493121126772390668778848955109301702439379995189437814430316454125759221675227357722242079059255648646310084394169874160200017558092442402438024383770066 [0xf26a4297000000000794864f00000000000000000000000019e8e934000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f90f6870ba06b193ce05b6f3fc99ce7da5ad48b66e4a4d5f19a90107a8ef47ae9148e7cdf99eaf068e6068cfd52a064c155b8601a1a0d25bae9a44850914aca63265582ea32db9ea817a4c80302db2aaf87ef9f651b5440ae330bfb16057cb5149be698f8f5dddfdd874c82aed99a92dd2] vs 6707390009636963749785842126143380118397061316769764293488516074335641302924033377090610142487812533856185953148490380715366375464631372553300548003929694617441663059443348887627101886807512307111989153120101947324785234665280121502907129173354190530417267577934568114388493121126772390668778848955109301702439379995189437814430316454125759221675227357722242079059255648646310084394169874160200017558092442402438024383770066 [0xf26a4297000000000794864f00000000000000000000000019e8e934000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f90f6870ba06b193ce05b6f3fc99ce7da5ad48b66e4a4d5f19a90107a8ef47ae9148e7cdf99eaf068e6068cfd52a064c155b8601a1a0d25bae9a44850914aca63265582ea32db9ea817a4c80302db2aaf87ef9f651b5440ae330bfb16057cb5149be698f8f5dddfdd874c82aed99a92dd2]) cdi_type: Attestation
  
   DiversificationKey act: 0x7ef9f651b5440ae330bfb16057cb5149be698f8f5dddfdd874c82aed99a92dd2, exp: 0x7ef9f651b5440ae330bfb16057cb5149be698f8f5dddfdd874c82aed99a92dd2
   RomDigest act: 0x5b8601a1a0d25bae9a44850914aca63265582ea32db9ea817a4c80302db2aaf8, exp: 0x5b8601a1a0d25bae9a44850914aca63265582ea32db9ea817a4c80302db2aaf8
   HealthMeasurement act: 0x48e7cdf99eaf068e6068cfd52a064c15, exp: 0x48e7cdf99eaf068e6068cfd52a064c15

... and 3 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 3 failures:
- Test keymgr_stress_all_with_rand_reset has 3 failures.
  - 17.keymgr_stress_all_with_rand_reset.2491219631
    Line 1055, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/17.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 617748001 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 617748001 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 21.keymgr_stress_all_with_rand_reset.3217553513
    Line 1594, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/21.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 208055276 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 208055276 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.

UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

24.keymgr_stress_all_with_rand_reset.3091194603
Line 1832, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 618551052 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2403149244 [0x8f3d25bc] vs 2403149244 [0x8f3d25bc])
  UVM_INFO @ 618551052 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

45.keymgr_stress_all_with_rand_reset.2242195303
Line 718, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/45.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @  67103487 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2248227312 [0x860139f0] vs 2248227312 [0x860139f0])
  UVM_INFO @  67103487 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1010) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 1 failures:

Test keymgr_stress_all_with_rand_reset has 1 failures.

33.keymgr_stress_all_with_rand_reset.2055346793
Line 887, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/33.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 201737342 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (5856635950747484500724999465842567023667514648376662579259962021934980621380177197345601748357250841277622592698355685002855701399395592429871591799893624181412691989560570305154901368066324944328012777774906405584772769485529989674961476738398923060784147755894133113527457296477329090597847245253339430099354372739383061158107670739242426260579996038809759870896975477965475586167414322674984038225299107913449919296957304 [0xd3aae515917158e374139e73811980bbe3d2752c24f1080597175b87c2d929093a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78] vs 3351901970189626218339616342014196247681054204505987745275814302926785129073393240628804421327414353073959263053830191071115368350062329176608283383502964889772730920500976287174481686116399288772459538847683099572011240772675763166585945638258442034050148551866711598890508924774810733184678174749678438314180277095001772901348073905151153633730396947085055695367827763566306648170032327658294915275430354002864113981188984 [0x79248a5e7b9e534b130ed8df386c20befd245adc317479ad0db43f919c83bd0e3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9000000000000000000000000000000000000000000000000000000000000f0f000000000000000005cfbd765ce33f34ea20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507fa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78]) cdi_type: Attestation
  
   DiversificationKey act: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78, exp: 0xfa365df52da48cd752fb3a026a8e608f0098cfe5fa9810494829d0cd9479eb78
   RomDigest act: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507, exp: 0xa20a046cf42e6eac560a3f82bfa76285b5c1d4aea7c915e49a32d1c89be0f507
   HealthMeasurement act: 0x5cfbd765ce33f34e, exp: 0x5cfbd765ce33f34e

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 47.keymgr_sync_async_fault_cross.173878207
    Line 230, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/47.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @   4089683 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   4089683 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```