KEYMGR Simulation Results

Saturday May 20 2023 07:05:26 UTC

GitHub Revision: `e3fb01b5e`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 2781625531

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	59.520s	3.778ms	50	50	100.00
V1	random	keymgr_random	1.330m	4.598ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.650s	612.177us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.180s	64.377us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	17.190s	1.340ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	17.040s	1.968ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.970s	25.702us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.180s	64.377us	20	20	100.00
		keymgr_csr_aliasing	17.040s	1.968ms	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	2.510m	2.931ms	50	50	100.00
V2	sideload	keymgr_sideload	1.345m	14.026ms	50	50	100.00
		keymgr_sideload_kmac	1.315m	11.650ms	50	50	100.00
		keymgr_sideload_aes	1.066m	3.611ms	50	50	100.00
		keymgr_sideload_otbn	1.288m	23.292ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	11.040s	354.082us	49	50	98.00
V2	lc_disable	keymgr_lc_disable	14.230s	925.350us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.257m	4.296ms	50	50	100.00
V2	invalid_sw_input	keymgr_sw_invalid_input	48.160s	3.337ms	49	50	98.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	29.450s	3.117ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	16.690s	592.772us	49	50	98.00
V2	stress_all	keymgr_stress_all	7.714m	46.051ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.970s	18.678us	50	50	100.00
V2	alert_test	keymgr_alert_test	0.960s	32.830us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.030s	836.750us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.030s	836.750us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.650s	612.177us	5	5	100.00
		keymgr_csr_rw	1.180s	64.377us	20	20	100.00
		keymgr_csr_aliasing	17.040s	1.968ms	5	5	100.00
		keymgr_same_csr_outstanding	4.200s	223.394us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.650s	612.177us	5	5	100.00
		keymgr_csr_rw	1.180s	64.377us	20	20	100.00
		keymgr_csr_aliasing	17.040s	1.968ms	5	5	100.00
		keymgr_same_csr_outstanding	4.200s	223.394us	20	20	100.00
V2		TOTAL			737	740	99.59
V2S	sec_cm_additional_check	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
		keymgr_tl_intg_err	11.370s	1.558ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	23.360s	2.742ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	23.360s	2.742ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	23.360s	2.742ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	23.360s	2.742ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.410s	1.657ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	11.370s	1.558ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	23.360s	2.742ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	2.510m	2.931ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.330m	4.598ms	50	50	100.00
		keymgr_csr_rw	1.180s	64.377us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.330m	4.598ms	50	50	100.00
		keymgr_csr_rw	1.180s	64.377us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.330m	4.598ms	50	50	100.00
		keymgr_csr_rw	1.180s	64.377us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	14.230s	925.350us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	29.450s	3.117ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	29.450s	3.117ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.330m	4.598ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	22.870s	1.883ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	29.210s	8.349ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	14.230s	925.350us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	29.210s	8.349ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	29.210s	8.349ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	29.210s	8.349ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	1.449m	7.436ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	29.210s	8.349ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	17.000s	547.683us	47	50	94.00
V3		TOTAL			47	50	94.00
		TOTAL			1104	1110	99.46

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	13	81.25
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.89	99.09	97.96	99.21	100.00	99.08	98.38	91.51

Failure Buckets

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 3 failures:

Test keymgr_direct_to_disabled has 1 failures.

28.keymgr_direct_to_disabled.2415682392
Line 347, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/28.keymgr_direct_to_disabled/latest/run.log

  UVM_ERROR @  27279148 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  27279148 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sw_invalid_input has 1 failures.

28.keymgr_sw_invalid_input.1707226339
Line 514, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/28.keymgr_sw_invalid_input/latest/run.log

  UVM_ERROR @  20991693 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  20991693 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_sync_async_fault_cross has 1 failures.

40.keymgr_sync_async_fault_cross.397855334
Line 236, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/40.keymgr_sync_async_fault_cross/latest/run.log

  UVM_ERROR @   7390811 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   7390811 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1010) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

2.keymgr_stress_all_with_rand_reset.1524671425
Line 979, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 394269283 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (1015388237085031103419536566272935776734508659955718273108561012884848142241668017381611449119392041435259668843928790394162581134781812825542943604427466031065975246502353211675562209284263314192143142449418273382956372265435925474759316777679626747454585237548704282617260331014667009316812581631160597031524793374500335460599551137239099760422230636561005777404999037487490678519246682727737306897921043137547337 [0x9d9d7cd93e378550c60d62cb00000000000000000bed329c5748dad23a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f97ce6517945bd0c7d0484db26142ab823296d96a1ba43c91f4321c54b232a561a1a295e9a3d80500c0d420210328342ca204ccf0cef49f8372cbdb2debf4fc9e108332e9bbb1e59bd2a69f6a486be365c0df3fa390302905238d5a0fe3bc29341c8c7477562bde4684b8f32a833b7cc49] vs 258005211985907388824662858662827450624945686551886006071429601413489389559739353390377624151907203134826876072856728629170473786894510329030062546543353126988216400795886582581189837705470544530708256838520663124740648601353953263013579331389129739810259305489686753595938075537210767013040727688796278375438218166114128719451893153557861414982790212107842323341145161189788168574957840609477658299321417 [0xac029e51e62287790000000000000000f35df656e1e881743a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f97ce6517945bd0c7d0484db26142ab823296d96a1ba43c91f4321c54b232a561a1a295e9a3d80500c0d420210328342ca204ccf0cef49f8372cbdb2debf4fc9e108332e9bbb1e59bd2a69f6a486be365c0df3fa390302905238d5a0fe3bc29341c8c7477562bde4684b8f32a833b7cc49]) cdi_type: Attestation
  
   DiversificationKey act: 0xdf3fa390302905238d5a0fe3bc29341c8c7477562bde4684b8f32a833b7cc49, exp: 0xdf3fa390302905238d5a0fe3bc29341c8c7477562bde4684b8f32a833b7cc49
   RomDigest act: 0x204ccf0cef49f8372cbdb2debf4fc9e108332e9bbb1e59bd2a69f6a486be365c, exp: 0x204ccf0cef49f8372cbdb2debf4fc9e108332e9bbb1e59bd2a69f6a486be365c
   HealthMeasurement act: 0x1a295e9a3d80500c0d420210328342ca, exp: 0x1a295e9a3d80500c0d420210328342ca

43.keymgr_stress_all_with_rand_reset.1447453224
Line 794, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/43.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 519183297 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (4077787920979729332414443991876246939784308934594435678771623033678006504218724228012255984445596027973810206113230252143183971390170584414180070155971976368927450438789678701325477953603510621071801516258304076757142567565138798677748905564221426115409979998764994755655826791407551896522767050227073332600774815817596768742568049114405269577761508619297438860269749643049751784779372595828838915159077520922172598808086188 [0x9360992c000000007b8e697aeee80fc01d0cd905000000006d42b4bb6a2b12443a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f970d4953b58bded266a264bb6d174b1e5a4a0fad28f62a0dc030bb1ced644329b1df563d408efcbf1a51100591ed1af7b155e4f561ee7cd0ffdf84cc4e701c052d42e8f6b870ecbecbbcc37a838a5384f3b9aa41290cae7f5a23ce61d1a1c296f2b6581cbe303dd374ea7ab256dce06ac] vs 4413938293704452986449172300409815888733662077387622223896249628167362519552718885339145753023628103050441397290440528603564623743991248835428424927179557516715073253876343104435645223270442534107242574906976994452294994926206678167315669246759993488860641394406432400868687190466716796824620171251609114950896622311689150865829293013767686082086885656807757687110776862960340600851718193218968677143095251523222872712414892 [0x9f86bc7373bf01023e075192855f0f2a00000000477122bb0000000032a9fe133a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f970d4953b58bded266a264bb6d174b1e5a4a0fad28f62a0dc030bb1ced644329b1df563d408efcbf1a51100591ed1af7b155e4f561ee7cd0ffdf84cc4e701c052d42e8f6b870ecbecbbcc37a838a5384f3b9aa41290cae7f5a23ce61d1a1c296f2b6581cbe303dd374ea7ab256dce06ac]) cdi_type: Attestation
  
   DiversificationKey act: 0x3b9aa41290cae7f5a23ce61d1a1c296f2b6581cbe303dd374ea7ab256dce06ac, exp: 0x3b9aa41290cae7f5a23ce61d1a1c296f2b6581cbe303dd374ea7ab256dce06ac
   RomDigest act: 0x155e4f561ee7cd0ffdf84cc4e701c052d42e8f6b870ecbecbbcc37a838a5384f, exp: 0x155e4f561ee7cd0ffdf84cc4e701c052d42e8f6b870ecbecbbcc37a838a5384f
   HealthMeasurement act: 0x1df563d408efcbf1a51100591ed1af7b, exp: 0x1df563d408efcbf1a51100591ed1af7b

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 15.keymgr_stress_all_with_rand_reset.2693962753
    Line 777, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/15.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 769343709 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 769343709 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```