KEYMGR Simulation Results

Monday February 26 2024 20:02:14 UTC

GitHub Revision: `9fb9dbc43f`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 749402974

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	50.910s	2.000ms	50	50	100.00
V1	random	keymgr_random	1.644m	8.367ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.410s	57.605us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.570s	266.675us	16	20	80.00
V1	csr_bit_bash	keymgr_csr_bit_bash	14.850s	999.716us	4	5	80.00
V1	csr_aliasing	keymgr_csr_aliasing	8.650s	511.826us	4	5	80.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.420s	117.594us	19	20	95.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.570s	266.675us	16	20	80.00
		keymgr_csr_aliasing	8.650s	511.826us	4	5	80.00
V1		TOTAL			148	155	95.48
V2	cfgen_during_op	keymgr_cfg_regwen	1.752m	7.691ms	50	50	100.00
V2	sideload	keymgr_sideload	1.224m	3.947ms	50	50	100.00
		keymgr_sideload_kmac	1.364m	4.584ms	50	50	100.00
		keymgr_sideload_aes	57.220s	5.171ms	50	50	100.00
		keymgr_sideload_otbn	33.990s	3.227ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	31.910s	3.409ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	17.080s	568.588us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.056m	6.543ms	44	50	88.00
V2	invalid_sw_input	keymgr_sw_invalid_input	55.340s	6.303ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.040m	1.895ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	22.030s	3.963ms	49	50	98.00
V2	stress_all	keymgr_stress_all	7.046m	42.803ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.870s	19.856us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.090s	25.934us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.860s	606.303us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.860s	606.303us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.410s	57.605us	5	5	100.00
		keymgr_csr_rw	1.570s	266.675us	16	20	80.00
		keymgr_csr_aliasing	8.650s	511.826us	4	5	80.00
		keymgr_same_csr_outstanding	3.780s	689.515us	12	20	60.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.410s	57.605us	5	5	100.00
		keymgr_csr_rw	1.570s	266.675us	16	20	80.00
		keymgr_csr_aliasing	8.650s	511.826us	4	5	80.00
		keymgr_same_csr_outstanding	3.780s	689.515us	12	20	60.00
V2		TOTAL			725	740	97.97
V2S	sec_cm_additional_check	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
		keymgr_tl_intg_err	11.140s	1.265ms	11	20	55.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	15.460s	869.041us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	15.460s	869.041us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	15.460s	869.041us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	15.460s	869.041us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	14.740s	1.627ms	14	20	70.00
V2S	prim_count_check	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	11.140s	1.265ms	11	20	55.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	15.460s	869.041us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.752m	7.691ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.644m	8.367ms	50	50	100.00
		keymgr_csr_rw	1.570s	266.675us	16	20	80.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.644m	8.367ms	50	50	100.00
		keymgr_csr_rw	1.570s	266.675us	16	20	80.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.644m	8.367ms	50	50	100.00
		keymgr_csr_rw	1.570s	266.675us	16	20	80.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	17.080s	568.588us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.040m	1.895ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.040m	1.895ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.644m	8.367ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	41.700s	9.594ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	18.300s	2.097ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	17.080s	568.588us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	18.300s	2.097ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	18.300s	2.097ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	18.300s	2.097ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	6.252m	40.358ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	18.300s	2.097ms	50	50	100.00
V2S		TOTAL			150	165	90.91
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	14.680s	187.854us	48	50	96.00
V3		TOTAL			48	50	96.00
		TOTAL			1071	1110	96.49

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	3	42.86
V2	16	16	13	81.25
V2S	6	6	4	66.67
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.83	99.10	97.95	98.58	100.00	99.11	98.41	91.63

Failure Buckets

Offending '(d2h.d_error || ((d2h.d_data & *) == (exp_vals[*] & *)))' has 30 failures:

Test keymgr_shadow_reg_errors_with_csr_rw has 6 failures.

0.keymgr_shadow_reg_errors_with_csr_rw.99403935
Line 261, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[12] & 'hffffffff)))'
  UVM_ERROR @  69872638 ps: (keymgr_csr_assert_fpv.sv:401) [ASSERT FAILED] sealing_sw_binding_7_rd_A
  UVM_INFO @  69872638 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

5.keymgr_shadow_reg_errors_with_csr_rw.912783471
Line 263, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_shadow_reg_errors_with_csr_rw/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[18] & 'hffffffff)))'
  UVM_ERROR @ 559071613 ps: (keymgr_csr_assert_fpv.sv:431) [ASSERT FAILED] attest_sw_binding_5_rd_A
  UVM_INFO @ 559071613 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 4 more failures.

Test keymgr_same_csr_outstanding has 8 failures.

0.keymgr_same_csr_outstanding.584545765
Line 259, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[5] & 'hffffffff)))'
  UVM_ERROR @  85677355 ps: (keymgr_csr_assert_fpv.sv:366) [ASSERT FAILED] sealing_sw_binding_0_rd_A
  UVM_INFO @  85677355 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

3.keymgr_same_csr_outstanding.1365423869
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_same_csr_outstanding/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[19] & 'hffffffff)))'
  UVM_ERROR @  11772205 ps: (keymgr_csr_assert_fpv.sv:436) [ASSERT FAILED] attest_sw_binding_6_rd_A
  UVM_INFO @  11772205 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 6 more failures.

Test keymgr_tl_intg_err has 9 failures.

1.keymgr_tl_intg_err.766020774
Line 262, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[6] & 'hffffffff)))'
  UVM_ERROR @  15854225 ps: (keymgr_csr_assert_fpv.sv:371) [ASSERT FAILED] sealing_sw_binding_1_rd_A
  UVM_INFO @  15854225 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

5.keymgr_tl_intg_err.328111127
Line 294, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_tl_intg_err/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[13] & 'hffffffff)))'
  UVM_ERROR @  58503167 ps: (keymgr_csr_assert_fpv.sv:406) [ASSERT FAILED] attest_sw_binding_0_rd_A
  UVM_INFO @  58503167 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 7 more failures.

Test keymgr_csr_bit_bash has 1 failures.

1.keymgr_csr_bit_bash.2932744560
Line 255, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_csr_bit_bash/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[19] & 'hffffffff)))'
  UVM_ERROR @ 1111008127 ps: (keymgr_csr_assert_fpv.sv:436) [ASSERT FAILED] attest_sw_binding_6_rd_A
  UVM_INFO @ 1111008127 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_csr_aliasing has 1 failures.

4.keymgr_csr_aliasing.963832160
Line 256, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_csr_aliasing/latest/run.log

      Offending '(d2h.d_error || ((d2h.d_data & 'hffffffff) == (exp_vals[7] & 'hffffffff)))'
  UVM_ERROR @ 892521920 ps: (keymgr_csr_assert_fpv.sv:376) [ASSERT FAILED] sealing_sw_binding_2_rd_A
  UVM_INFO @ 892521920 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more tests.

UVM_ERROR (keymgr_scoreboard.sv:1019) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 6 failures:

Test keymgr_kmac_rsp_err has 6 failures.

0.keymgr_kmac_rsp_err.1353073945
Line 579, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 284886347 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (803608758794043356075870555956147438455094219993510862635962330599923369757649716267961935464335957066642670596298415480562132231947869314235210117454261103755589979292374001505535408747666247944157489347610904502002783321307560963652350646225352290587563769268985042058717681327365235355472544460586865832019418175080732193451625024665828840234086770144679848565827585197894508811136019246405352470351383811290986506448348616614242976259095901650651473606105968232925245536804551826763118698755109280 [0xfad359d9bd5643271f219af85731b3d11328f5a5ab970044e0eb4b28a804abcb3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9fe883b910987a213e5bb8702295838c42c8b64d53688a98fc380083e4b0798371fada610816121e9c2461726ebecf80a1d2e3e475d782ed7c1f24a5044bf9d1b1a3e147b9f9aeab5b6f71ab8134c283e38759c6ac6f895118d5788f0bf9351d2d62aab304ae9649a845b64d11a386bc0ec705ab9ca85a137040e2458394611de6792044e1f5fdb76d9ece59ab7c9b1a0] vs 803608758794043356075870555956147438455094219993510862635962330599923369757649716267961935464335957066642670596298415480562132231947869314235210117454261103755589979292374001505535408747666247944157489347610904502002783321307560963652350646225352290587563769268985042058717681327365235355472544460586865832019418175080732193451625024665828840234086770144679848565827585197894508811136019246405352470351383811290986506448348616614242976259095901650651473606105968232925245536804551826763118698755109280 [0xfad359d9bd5643271f219af85731b3d11328f5a5ab970044e0eb4b28a804abcb3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9fe883b910987a213e5bb8702295838c42c8b64d53688a98fc380083e4b0798371fada610816121e9c2461726ebecf80a1d2e3e475d782ed7c1f24a5044bf9d1b1a3e147b9f9aeab5b6f71ab8134c283e38759c6ac6f895118d5788f0bf9351d2d62aab304ae9649a845b64d11a386bc0ec705ab9ca85a137040e2458394611de6792044e1f5fdb76d9ece59ab7c9b1a0]) cdi_type: Attestation
  
   DiversificationKey act: 0xec705ab9ca85a137040e2458394611de6792044e1f5fdb76d9ece59ab7c9b1a0, exp: 0xec705ab9ca85a137040e2458394611de6792044e1f5fdb76d9ece59ab7c9b1a0
   RomDigests act: 0x1d2e3e475d782ed7c1f24a5044bf9d1b1a3e147b9f9aeab5b6f71ab8134c283e38759c6ac6f895118d5788f0bf9351d2d62aab304ae9649a845b64d11a386bc0, exp: 0x1d2e3e475d782ed7c1f24a5044bf9d1b1a3e147b9f9aeab5b6f71ab8134c283e38759c6ac6f895118d5788f0bf9351d2d62aab304ae9649a845b64d11a386bc0
   HealthMeasurement act: 0x1fada610816121e9c2461726ebecf80a, exp: 0x1fada610816121e9c2461726ebecf80a

12.keymgr_kmac_rsp_err.1915480421
Line 364, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/12.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 236449154 ps: (keymgr_scoreboard.sv:1019) [uvm_test_top.env.scoreboard] Check failed act != exp (56030937028044239692298131968177709160202486393126974606821070932022639672389726304526766015030561888470780162913550261881466158249255824532230659164146832764007574708182886578028923353585854296813194264228758790785441610503365300225276587623659474833644370878207085519131377452431842151356331055088783612494528988878382804837105604507818523425220439265542665371451463517949125729681444338636153795243865227569530215303114430787140575349333419388679769052368999721461958127456906192634068523625000108 [0x117d14f20e68d2340785d631f061f3da79373cb8a47c36b3ec231682fad1974a3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f944d9a9abd177ad57bc55305b09ffb3fdbb4100d1c02f27de9f16c5eebd2c907e99a412b46eb59182ae64219e72c1804fcd4425f358450386ccc2ab1f4717797d76765c9e760958ea05f5af978fc3c7af53e863fd11a06f98d81f3670f7042eaa16d5952c1aae1d6e54473204b56aa6e83035eff5fcc88be5817030745192ad43017f3a628c324aa2f7ef61a5472cacac] vs 56030937028044239692298131968177709160202486393126974606821070932022639672389726304526766015030561888470780162913550261881466158249255824532230659164146832764007574708182886578028923353585854296813194264228758790785441610503365300225276587623659474833644370878207085519131377452431842151356331055088783612494528988878382804837105604507818523425220439265542665371451463517949125729681444338636153795243865227569530215303114430787140575349333419388679769052368999721461958127456906192634068523625000108 [0x117d14f20e68d2340785d631f061f3da79373cb8a47c36b3ec231682fad1974a3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f944d9a9abd177ad57bc55305b09ffb3fdbb4100d1c02f27de9f16c5eebd2c907e99a412b46eb59182ae64219e72c1804fcd4425f358450386ccc2ab1f4717797d76765c9e760958ea05f5af978fc3c7af53e863fd11a06f98d81f3670f7042eaa16d5952c1aae1d6e54473204b56aa6e83035eff5fcc88be5817030745192ad43017f3a628c324aa2f7ef61a5472cacac]) cdi_type: Attestation
  
   DiversificationKey act: 0x3035eff5fcc88be5817030745192ad43017f3a628c324aa2f7ef61a5472cacac, exp: 0x3035eff5fcc88be5817030745192ad43017f3a628c324aa2f7ef61a5472cacac
   RomDigests act: 0xcd4425f358450386ccc2ab1f4717797d76765c9e760958ea05f5af978fc3c7af53e863fd11a06f98d81f3670f7042eaa16d5952c1aae1d6e54473204b56aa6e8, exp: 0xcd4425f358450386ccc2ab1f4717797d76765c9e760958ea05f5af978fc3c7af53e863fd11a06f98d81f3670f7042eaa16d5952c1aae1d6e54473204b56aa6e8
   HealthMeasurement act: 0x99a412b46eb59182ae64219e72c1804f, exp: 0x99a412b46eb59182ae64219e72c1804f

... and 4 more failures.

UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 2 failures:

Test keymgr_stress_all_with_rand_reset has 2 failures.

19.keymgr_stress_all_with_rand_reset.3362068912
Line 977, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/19.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 146012718 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (1007006256 [0x3c05b230] vs 1007006256 [0x3c05b230])
  UVM_INFO @ 146012718 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

44.keymgr_stress_all_with_rand_reset.564436866
Line 728, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/44.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 144550580 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (1711025528 [0x65fc2d78] vs 1711025528 [0x65fc2d78])
  UVM_INFO @ 144550580 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_scoreboard.sv:241) scoreboard [scoreboard] alert recov_operation_err is not received! has 1 failures:
- Test keymgr_sync_async_fault_cross has 1 failures.
  - 21.keymgr_sync_async_fault_cross.1097400506
    Line 382, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/21.keymgr_sync_async_fault_cross/latest/run.log
```
  UVM_ERROR @ 282870907 ps: (cip_base_scoreboard.sv:241) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err is not received!
  UVM_INFO @ 282870907 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```