KEYMGR Simulation Results

Tuesday July 30 2024 23:02:08 UTC

GitHub Revision: `fdfa12db04`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 101467584611478134588291649782725219255540557286164709436567235390830780957271

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	39.380s	10.271ms	50	50	100.00
V1	random	keymgr_random	1.438m	2.730ms	49	50	98.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.640s	37.472us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.570s	63.016us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	18.060s	1.324ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	9.030s	375.852us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.830s	38.718us	19	20	95.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.570s	63.016us	20	20	100.00
		keymgr_csr_aliasing	9.030s	375.852us	5	5	100.00
V1		TOTAL			153	155	98.71
V2	cfgen_during_op	keymgr_cfg_regwen	1.753m	4.112ms	48	50	96.00
V2	sideload	keymgr_sideload	28.710s	4.368ms	50	50	100.00
		keymgr_sideload_kmac	1.207m	7.121ms	50	50	100.00
		keymgr_sideload_aes	55.030s	6.074ms	50	50	100.00
		keymgr_sideload_otbn	1.248m	8.011ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	30.710s	2.551ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	15.130s	2.257ms	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	38.560s	4.065ms	50	50	100.00
V2	invalid_sw_input	keymgr_sw_invalid_input	57.230s	2.179ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	38.930s	10.075ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	23.270s	1.547ms	50	50	100.00
V2	stress_all	keymgr_stress_all	6.473m	24.910ms	49	50	98.00
V2	intr_test	keymgr_intr_test	0.940s	12.074us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.010s	31.161us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	5.620s	515.240us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	5.620s	515.240us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.640s	37.472us	5	5	100.00
		keymgr_csr_rw	1.570s	63.016us	20	20	100.00
		keymgr_csr_aliasing	9.030s	375.852us	5	5	100.00
		keymgr_same_csr_outstanding	3.970s	773.396us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.640s	37.472us	5	5	100.00
		keymgr_csr_rw	1.570s	63.016us	20	20	100.00
		keymgr_csr_aliasing	9.030s	375.852us	5	5	100.00
		keymgr_same_csr_outstanding	3.970s	773.396us	20	20	100.00
V2		TOTAL			737	740	99.59
V2S	sec_cm_additional_check	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
		keymgr_tl_intg_err	8.510s	225.124us	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	3.870s	173.864us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	3.870s	173.864us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	3.870s	173.864us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	3.870s	173.864us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	15.630s	1.557ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	8.510s	225.124us	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	3.870s	173.864us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.753m	4.112ms	48	50	96.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.438m	2.730ms	49	50	98.00
		keymgr_csr_rw	1.570s	63.016us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.438m	2.730ms	49	50	98.00
		keymgr_csr_rw	1.570s	63.016us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.438m	2.730ms	49	50	98.00
		keymgr_csr_rw	1.570s	63.016us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	15.130s	2.257ms	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	38.930s	10.075ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	38.930s	10.075ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.438m	2.730ms	49	50	98.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	18.000s	1.530ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	14.640s	1.434ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	15.130s	2.257ms	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	14.640s	1.434ms	50	50	100.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	14.640s	1.434ms	50	50	100.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	14.640s	1.434ms	50	50	100.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	20.470s	986.623us	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	14.640s	1.434ms	50	50	100.00
V2S		TOTAL			165	165	100.00
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	33.290s	6.177ms	29	50	58.00
V3		TOTAL			29	50	58.00
		TOTAL			1084	1110	97.66

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	5	71.43
V2	16	16	14	87.50
V2S	6	6	6	100.00
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.74	99.04	98.11	98.40	100.00	99.02	98.41	91.22

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:836) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 18 failures:
- Test keymgr_stress_all_with_rand_reset has 18 failures.
  - 5.keymgr_stress_all_with_rand_reset.38223503565566282608242518743574025860870884975933458019956200414327299476537
    Line 371, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 120507525 ps: (cip_base_vseq.sv:836) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10002 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 120507525 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 6.keymgr_stress_all_with_rand_reset.29110693794557409109394904861915083660222667653651492560776409529820615460311
    Line 466, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 2029606564 ps: (cip_base_vseq.sv:836) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 2029606564 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 16 more failures.
UVM_ERROR (cip_base_scoreboard.sv:301) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 4 failures:
- Test keymgr_cfg_regwen has 1 failures.
  - 14.keymgr_cfg_regwen.112331534500510969814833239315788459194953910166546485649303113506252749879476
    Line 384, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @  20153999 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  20153999 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 24.keymgr_stress_all_with_rand_reset.49167397450771207615350563015598361351005907064867855815567789348204363888892
    Line 826, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 102614669 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 102614669 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_random has 1 failures.
  - 32.keymgr_random.82813591044069698978105394460425266971987030695680816209965026982327239396083
    Line 268, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/32.keymgr_random/latest/run.log
```
  UVM_ERROR @  10300251 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  10300251 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_stress_all has 1 failures.
  - 41.keymgr_stress_all.112465880259242127611276955613198391158119621754325595259721939741903588511178
    Line 370, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/41.keymgr_stress_all/latest/run.log
```
  UVM_ERROR @  26505594 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  26505594 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (csr_utils_pkg.sv:476) [csr_utils::csr_rd_check] Check failed obs == exp (* [*] vs * [*]) Regname: keymgr_reg_block.sw_binding_regwen reset value: * has 1 failures:
- Test keymgr_csr_mem_rw_with_rand_reset has 1 failures.
  - 11.keymgr_csr_mem_rw_with_rand_reset.87922008732279873581546390652619873121707725815440941037964650305835612724421
    Line 269, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/11.keymgr_csr_mem_rw_with_rand_reset/latest/run.log
```
  UVM_ERROR @   7996277 ps: (csr_utils_pkg.sv:476) [csr_utils::csr_rd_check] Check failed obs == exp (0 [0x0] vs 1 [0x1]) Regname: keymgr_reg_block.sw_binding_regwen reset value: 0x1
  UVM_INFO @   7996277 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:692) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 19.keymgr_stress_all_with_rand_reset.48822433756516633256928448853977931688961410567545452957170993890743584790745
    Line 829, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/19.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 442743731 ps: (keymgr_scoreboard.sv:692) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 442743731 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:794) [scoreboard] Check failed item.d_data ==gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.start` has 1 failures:
- Test keymgr_cfg_regwen has 1 failures.
  - 29.keymgr_cfg_regwen.95064078327720916990295022523213412481457654257682679327228521702647714603649
    Line 321, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/29.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @  12074712 ps: (keymgr_scoreboard.sv:794) [uvm_test_top.env.scoreboard] Check failed item.d_data == `gmv(csr) (0 [0x0] vs 1 [0x1]) reg name: keymgr_reg_block.start
  UVM_INFO @  12074712 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (keymgr_scoreboard.sv:766) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * []) reg name: keymgr_reg_block.sw_share1_output_*` has 1 failures:
- Test keymgr_stress_all_with_rand_reset has 1 failures.
  - 38.keymgr_stress_all_with_rand_reset.24912527084828594760082712939506615034322178318425155241082764000267796850495
    Line 1816, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/38.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 434878160 ps: (keymgr_scoreboard.sv:766) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (0 [0x0] vs 0 [0x0]) reg name: keymgr_reg_block.sw_share1_output_2
  UVM_INFO @ 434878160 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

KEYMGR Simulation Results

Tuesday July 30 2024 23:02:08 UTC

GitHub Revision: fdfa12db04

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 101467584611478134588291649782725219255540557286164709436567235390830780957271

Test Results

Testplan Progress

Coverage Results

Coverage Dashboard

Failure Buckets

Past Results

GitHub Revision: `fdfa12db04`