KEYMGR Simulation Results

Wednesday July 24 2024 23:04:46 UTC

GitHub Revision: `e439226b6c`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 80778109121175195808319778278610424989650974127729484509360263424111433728567

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	30.780s	1.096ms	50	50	100.00
V1	random	keymgr_random	46.320s	2.641ms	49	50	98.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.230s	31.612us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.780s	44.919us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	14.760s	250.475us	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	14.930s	415.392us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.430s	52.055us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.780s	44.919us	20	20	100.00
		keymgr_csr_aliasing	14.930s	415.392us	5	5	100.00
V1		TOTAL			154	155	99.35
V2	cfgen_during_op	keymgr_cfg_regwen	1.303m	1.542ms	50	50	100.00
V2	sideload	keymgr_sideload	36.900s	2.695ms	50	50	100.00
		keymgr_sideload_kmac	37.640s	3.344ms	50	50	100.00
		keymgr_sideload_aes	1.034m	7.029ms	50	50	100.00
		keymgr_sideload_otbn	53.500s	24.556ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	33.760s	4.661ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	6.720s	337.690us	48	50	96.00
V2	kmac_error_response	keymgr_kmac_rsp_err	9.120s	464.372us	49	50	98.00
V2	invalid_sw_input	keymgr_sw_invalid_input	52.930s	13.092ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	52.660s	8.922ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	11.660s	1.063ms	50	50	100.00
V2	stress_all	keymgr_stress_all	3.044m	7.635ms	46	50	92.00
V2	intr_test	keymgr_intr_test	0.910s	41.235us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.020s	20.580us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.330s	1.469ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.330s	1.469ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.230s	31.612us	5	5	100.00
		keymgr_csr_rw	1.780s	44.919us	20	20	100.00
		keymgr_csr_aliasing	14.930s	415.392us	5	5	100.00
		keymgr_same_csr_outstanding	4.450s	1.168ms	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.230s	31.612us	5	5	100.00
		keymgr_csr_rw	1.780s	44.919us	20	20	100.00
		keymgr_csr_aliasing	14.930s	415.392us	5	5	100.00
		keymgr_same_csr_outstanding	4.450s	1.168ms	20	20	100.00
V2		TOTAL			733	740	99.05
V2S	sec_cm_additional_check	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
		keymgr_tl_intg_err	9.290s	1.176ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	4.840s	160.523us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	4.840s	160.523us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	4.840s	160.523us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	4.840s	160.523us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	14.390s	407.129us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	9.290s	1.176ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	4.840s	160.523us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.303m	1.542ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	46.320s	2.641ms	49	50	98.00
		keymgr_csr_rw	1.780s	44.919us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	46.320s	2.641ms	49	50	98.00
		keymgr_csr_rw	1.780s	44.919us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	46.320s	2.641ms	49	50	98.00
		keymgr_csr_rw	1.780s	44.919us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	6.720s	337.690us	48	50	96.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	52.660s	8.922ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	52.660s	8.922ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	46.320s	2.641ms	49	50	98.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	20.660s	822.680us	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	29.340s	3.807ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	6.720s	337.690us	48	50	96.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	29.340s	3.807ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	29.340s	3.807ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	29.340s	3.807ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	14.180s	660.011us	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	29.340s	3.807ms	49	50	98.00
V2S		TOTAL			164	165	99.39
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	28.840s	2.265ms	28	50	56.00
V3		TOTAL			28	50	56.00
		TOTAL			1079	1110	97.21

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	6	85.71
V2	16	16	13	81.25
V2S	6	6	5	83.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.72	99.04	97.95	98.53	100.00	99.02	98.41	91.07

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:839) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 21 failures:
- Test keymgr_stress_all_with_rand_reset has 21 failures.
  - 0.keymgr_stress_all_with_rand_reset.43342193064451834672047007636487707612270666530406324475942111610126669344260
    Line 841, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/0.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 515841988 ps: (cip_base_vseq.sv:839) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 515841988 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 4.keymgr_stress_all_with_rand_reset.47650344850685091701371809208166943252288521935814271010856916207391438752543
    Line 629, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/4.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1002097756 ps: (cip_base_vseq.sv:839) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 1002097756 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 19 more failures.

UVM_ERROR (cip_base_scoreboard.sv:301) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 6 failures:

Test keymgr_kmac_rsp_err has 1 failures.

7.keymgr_kmac_rsp_err.42226288926547707084090181641494216453632496091353511256663438068195676152791
Line 393, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/7.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @   7115561 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   7115561 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 2 failures.

23.keymgr_stress_all.80485917571552313614366540599110179762776287533688401853497275126884991339004
Line 2861, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/23.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 560573281 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 560573281 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

36.keymgr_stress_all.107871564161578673303025933267061463958684021537678135708394733184325753200129
Line 2093, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/36.keymgr_stress_all/latest/run.log

  UVM_ERROR @ 4352461685 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 4352461685 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_custom_cm has 1 failures.

29.keymgr_custom_cm.56974904400795837841204484200498999719931011030045506277413415190038750305639
Line 262, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/29.keymgr_custom_cm/latest/run.log

  UVM_ERROR @   5986766 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @   5986766 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all_with_rand_reset has 1 failures.
- 29.keymgr_stress_all_with_rand_reset.973150365224983780595689404288802973052838357947979117170816367559009607012
  Line 541, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/29.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @  48304326 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  48304326 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

Test keymgr_random has 1 failures.

33.keymgr_random.45164593119543275277106362697563881895864400357724152879718933379579772599894
Line 377, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/33.keymgr_random/latest/run.log

  UVM_ERROR @  23032144 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  23032144 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:766) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 1 failures:
- Test keymgr_lc_disable has 1 failures.
  - 3.keymgr_lc_disable.103479952439004499304031314325527385694770423455767307235618440205637794818498
    Line 307, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_lc_disable/latest/run.log
```
  UVM_ERROR @  39737232 ps: (keymgr_scoreboard.sv:766) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (0 [0x0] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_5
  UVM_INFO @  39737232 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:499) [scoreboard] Check failed item.d_data[keymgr_pkg::ErrInvalidOp] == err_code[keymgr_pkg::ErrInvalidOp] (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 39.keymgr_stress_all.48318869664036795317460322662573176298476936795988539386626360532136515818988
    Line 2631, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/39.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 1765810390 ps: (keymgr_scoreboard.sv:499) [uvm_test_top.env.scoreboard] Check failed item.d_data[keymgr_pkg::ErrInvalidOp] == err_code[keymgr_pkg::ErrInvalidOp] (0 [0x0] vs 1 [0x1])
  UVM_INFO @ 1765810390 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:766) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share0_output_*` has 1 failures:
- Test keymgr_stress_all has 1 failures.
  - 47.keymgr_stress_all.80401360761242331620345261328385911961840704065969791224256735786588471533566
    Line 3828, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/47.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @ 2862542367 ps: (keymgr_scoreboard.sv:766) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (0 [0x0] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share0_output_1
  UVM_INFO @ 2862542367 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

UVM_ERROR (keymgr_if.sv:557) [keymgr_if] Check failed {act_key.key[*], act_key.key[*]} !== keys_a_array[state][cdi][dest] (* [*] vs * [*]) AES key at state StCreatorRootKey for Sealing Aes has 1 failures:

Test keymgr_lc_disable has 1 failures.

49.keymgr_lc_disable.91573875631185623817305496052641141865602083434636246822143277181011492432186
Line 467, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/49.keymgr_lc_disable/latest/run.log

  UVM_ERROR @  25521718 ps: (keymgr_if.sv:557) [keymgr_if] Check failed {act_key.key[1], act_key.key[0]} !== keys_a_array[state][cdi][dest] (1428631494037108595052642318138437113828579794284332172400881998758788765532550889146604675582198701514444875795729726786930958157941672254117704659086691 [0x1b47015a273c0485b10fd96e8696cbc409eff2a41338d418490709a982a00109bc2851cfda3d498cde86887afe289f6dfaa5f36538707513c513dfd7e2f7ad63] vs 1428631494037108595052642318138437113828579794284332172400881998758788765532550889146604675582198701514444875795729726786930958157941672254117704659086691 [0x1b47015a273c0485b10fd96e8696cbc409eff2a41338d418490709a982a00109bc2851cfda3d498cde86887afe289f6dfaa5f36538707513c513dfd7e2f7ad63]) AES key at state StCreatorRootKey for Sealing Aes
  UVM_INFO @  25521718 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---