KEYMGR Simulation Results

Sunday March 03 2024 20:02:47 UTC

GitHub Revision: `0cdf265eaa`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 82530437672810453765703374940713112405319051694331588453064008042331386550559

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	1.222m	9.210ms	50	50	100.00
V1	random	keymgr_random	1.578m	9.413ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.240s	127.864us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.520s	25.430us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	17.420s	12.166ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	10.350s	363.029us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.290s	52.090us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.520s	25.430us	20	20	100.00
		keymgr_csr_aliasing	10.350s	363.029us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	2.146m	4.720ms	50	50	100.00
V2	sideload	keymgr_sideload	43.670s	1.547ms	50	50	100.00
		keymgr_sideload_kmac	1.093m	27.111ms	50	50	100.00
		keymgr_sideload_aes	28.030s	1.166ms	50	50	100.00
		keymgr_sideload_otbn	45.160s	4.915ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	39.910s	1.866ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	25.940s	502.863us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.494m	19.031ms	46	50	92.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.152m	2.782ms	49	50	98.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.160m	2.605ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	47.540s	12.674ms	49	50	98.00
V2	stress_all	keymgr_stress_all	5.125m	14.302ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.920s	23.537us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.070s	73.748us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	6.040s	366.354us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	6.040s	366.354us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.240s	127.864us	5	5	100.00
		keymgr_csr_rw	1.520s	25.430us	20	20	100.00
		keymgr_csr_aliasing	10.350s	363.029us	5	5	100.00
		keymgr_same_csr_outstanding	3.100s	168.617us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.240s	127.864us	5	5	100.00
		keymgr_csr_rw	1.520s	25.430us	20	20	100.00
		keymgr_csr_aliasing	10.350s	363.029us	5	5	100.00
		keymgr_same_csr_outstanding	3.100s	168.617us	20	20	100.00
V2		TOTAL			734	740	99.19
V2S	sec_cm_additional_check	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
		keymgr_tl_intg_err	59.140s	2.555ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	22.450s	1.063ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	22.450s	1.063ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	22.450s	1.063ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	22.450s	1.063ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.310s	1.687ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	59.140s	2.555ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	22.450s	1.063ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.146m	4.720ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.578m	9.413ms	50	50	100.00
		keymgr_csr_rw	1.520s	25.430us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.578m	9.413ms	50	50	100.00
		keymgr_csr_rw	1.520s	25.430us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.578m	9.413ms	50	50	100.00
		keymgr_csr_rw	1.520s	25.430us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	25.940s	502.863us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.160m	2.605ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.160m	2.605ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.578m	9.413ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	34.970s	1.166ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	20.140s	2.829ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	25.940s	502.863us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	20.140s	2.829ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	20.140s	2.829ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	20.140s	2.829ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	3.110m	33.571ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	20.140s	2.829ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	26.140s	4.885ms	20	50	40.00
V3		TOTAL			20	50	40.00
		TOTAL			1074	1110	96.76

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	13	81.25
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.85	99.07	98.14	98.72	100.00	99.11	98.41	91.54

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:815) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 26 failures:
- Test keymgr_stress_all_with_rand_reset has 26 failures.
  - 2.keymgr_stress_all_with_rand_reset.38622367864465185349657639414479886395984232069112574235459295524974656504130
    Line 271, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 426890679 ps: (cip_base_vseq.sv:815) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10001 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 426890679 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 4.keymgr_stress_all_with_rand_reset.72647541656407429003674939069491743396650711872070642156270891915727343386494
    Line 530, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 546155563 ps: (cip_base_vseq.sv:815) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 546155563 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 24 more failures.
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 4 failures:
- Test keymgr_stress_all_with_rand_reset has 4 failures.
  - 10.keymgr_stress_all_with_rand_reset.77317094257880158749814028344484080583423888737051878050031468020386279464826
    Line 1822, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1110014793 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (2853625426 [0xaa16de52] vs 2853625426 [0xaa16de52])
  UVM_INFO @ 1110014793 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 19.keymgr_stress_all_with_rand_reset.112853380276066985910861475181569676646510832700257329570839527266754447988721
    Line 1700, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/19.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1393384860 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (427133404 [0x197589dc] vs 427133404 [0x197589dc])
  UVM_INFO @ 1393384860 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 2 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1013) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 3 failures:

Test keymgr_kmac_rsp_err has 3 failures.

9.keymgr_kmac_rsp_err.19267559153990723039097562161786564550869868897086164793350698550873857906824
Line 350, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/9.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 218726026 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (4927172818047670749426230068164284411407258925782050236226568451867966082068897508734078692841020836322540427986007513325309958338149189479424106041982843974416405323775103737164509049798385274693593072331104312710708378293728879097784504957270342236242784720613005160484477014418808296703878837027466965147394376776420959852190965110890645807415425288107585759721107686900506384349351079149368834330140165296423533883390036 [0xb2134bb93890333a524e7b4eab8ccec76fb3b706baebe5024b7075db71fd55893a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9997f7d25b6ae77177470c41e2dce545b3c67f55cff62836fc5f2eaa2e2732c781f068ee43f9b351b7f494ed6a2586614f595b7f18664f37367b58bafa289dfc2428490d06025feb22ec902919510236e028c667da9856d423531e112d5949528bb4efbe1a3684f895291575e2d237854] vs 4927172818047670749426230068164284411407258925782050236226568451867966082068897508734078692841020836322540427986007513325309958338149189479424106041982843974416405323775103737164509049798385274693593072331104312710708378293728879097784504957270342236242784720613005160484477014418808296703878837027466965147394376776420959852190965110890645807415425288107585759721107686900506384349351079149368834330140165296423533883390036 [0xb2134bb93890333a524e7b4eab8ccec76fb3b706baebe5024b7075db71fd55893a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9997f7d25b6ae77177470c41e2dce545b3c67f55cff62836fc5f2eaa2e2732c781f068ee43f9b351b7f494ed6a2586614f595b7f18664f37367b58bafa289dfc2428490d06025feb22ec902919510236e028c667da9856d423531e112d5949528bb4efbe1a3684f895291575e2d237854]) cdi_type: Attestation
  
   DiversificationKey act: 0x28c667da9856d423531e112d5949528bb4efbe1a3684f895291575e2d237854, exp: 0x28c667da9856d423531e112d5949528bb4efbe1a3684f895291575e2d237854
   RomDigest act: 0xf595b7f18664f37367b58bafa289dfc2428490d06025feb22ec902919510236e, exp: 0xf595b7f18664f37367b58bafa289dfc2428490d06025feb22ec902919510236e
   HealthMeasurement act: 0x1f068ee43f9b351b7f494ed6a2586614, exp: 0x1f068ee43f9b351b7f494ed6a2586614

13.keymgr_kmac_rsp_err.75785360205872446677609072901577230502807625898349576878926268523592481482963
Line 497, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/13.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  50489182 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (571104933094790195038198212445520555824514738926562594713065118764488020382795348259465370154184550747323455567529482124041954692266037987575426752357312911326375450111250874139538531877859985267096725191272714341584197856179341357534386347487901360726594725883546687793954309753580701378981481770041543638889212274522639184748892024071158631219262889161080869856249692838375893712194711767862390425726290002686043023751925 [0x14a3fd50eeb2f7dfd4ac900d7a6d32b06ca82a7a815f4ea612fcc16a42fa584d3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f95f6a0fdb0322e32cc1cef064a204a11cd1eb0b83dd2099d8cafb437c54ff71fadf6fc9e507131ea98dc9cfa570cdbe98dd01b83be61c277db5a7d5d8018e8af804be16ba38d4c81b0910bdd10184bf13e88be8a90051d982bf88c981d5de0e735791b43534cc498569cd550d52503af5] vs 571104933094790195038198212445520555824514738926562594713065118764488020382795348259465370154184550747323455567529482124041954692266037987575426752357312911326375450111250874139538531877859985267096725191272714341584197856179341357534386347487901360726594725883546687793954309753580701378981481770041543638889212274522639184748892024071158631219262889161080869856249692838375893712194711767862390425726290002686043023751925 [0x14a3fd50eeb2f7dfd4ac900d7a6d32b06ca82a7a815f4ea612fcc16a42fa584d3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f95f6a0fdb0322e32cc1cef064a204a11cd1eb0b83dd2099d8cafb437c54ff71fadf6fc9e507131ea98dc9cfa570cdbe98dd01b83be61c277db5a7d5d8018e8af804be16ba38d4c81b0910bdd10184bf13e88be8a90051d982bf88c981d5de0e735791b43534cc498569cd550d52503af5]) cdi_type: Attestation
  
   DiversificationKey act: 0xe88be8a90051d982bf88c981d5de0e735791b43534cc498569cd550d52503af5, exp: 0xe88be8a90051d982bf88c981d5de0e735791b43534cc498569cd550d52503af5
   RomDigest act: 0xdd01b83be61c277db5a7d5d8018e8af804be16ba38d4c81b0910bdd10184bf13, exp: 0xdd01b83be61c277db5a7d5d8018e8af804be16ba38d4c81b0910bdd10184bf13
   HealthMeasurement act: 0xdf6fc9e507131ea98dc9cfa570cdbe98, exp: 0xdf6fc9e507131ea98dc9cfa570cdbe98

... and 1 more failures.

UVM_ERROR (cip_base_scoreboard.sv:287) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 2 failures:
- Test keymgr_sw_invalid_input has 1 failures.
  - 28.keymgr_sw_invalid_input.48981021746159502327193608167840660960086288521265976169175237076489724645146
    Line 345, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/28.keymgr_sw_invalid_input/latest/run.log
```
  UVM_ERROR @  16469130 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  16469130 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_kmac_rsp_err has 1 failures.
  - 36.keymgr_kmac_rsp_err.50191669719936987901253889975980184343724379208623857372733407885269477616791
    Line 320, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_kmac_rsp_err/latest/run.log
```
  UVM_ERROR @  24896289 ps: (cip_base_scoreboard.sv:287) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  24896289 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_scoreboard.sv:247) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 43.keymgr_sync_async_fault_cross.17798752185692511621971957858514409137123244187405297840818747551538028338933
    Line 343, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/43.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 124548703 ps: (cip_base_scoreboard.sv:247) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 124548703 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```