KEYMGR Simulation Results

Tuesday April 02 2024 19:02:21 UTC

GitHub Revision: `1fbe1ece8d`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 10515816417091650402163962333134174777740454699264757911298152460288222033634

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	52.160s	6.260ms	50	50	100.00
V1	random	keymgr_random	1.158m	4.852ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.230s	65.933us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.560s	114.687us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	24.140s	12.762ms	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	10.120s	762.402us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	2.240s	741.676us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.560s	114.687us	20	20	100.00
		keymgr_csr_aliasing	10.120s	762.402us	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	59.430s	2.294ms	49	50	98.00
V2	sideload	keymgr_sideload	38.520s	2.936ms	50	50	100.00
		keymgr_sideload_kmac	1.005m	7.721ms	50	50	100.00
		keymgr_sideload_aes	1.114m	1.971ms	50	50	100.00
		keymgr_sideload_otbn	47.960s	1.658ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	19.190s	3.130ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	7.610s	538.643us	50	50	100.00
V2	kmac_error_response	keymgr_kmac_rsp_err	54.160s	1.899ms	43	50	86.00
V2	invalid_sw_input	keymgr_sw_invalid_input	58.630s	1.642ms	49	50	98.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.359m	5.073ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	26.610s	1.071ms	50	50	100.00
V2	stress_all	keymgr_stress_all	7.030m	87.248ms	50	50	100.00
V2	intr_test	keymgr_intr_test	0.950s	17.442us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.110s	27.252us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	6.370s	386.794us	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	6.370s	386.794us	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.230s	65.933us	5	5	100.00
		keymgr_csr_rw	1.560s	114.687us	20	20	100.00
		keymgr_csr_aliasing	10.120s	762.402us	5	5	100.00
		keymgr_same_csr_outstanding	4.290s	119.830us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.230s	65.933us	5	5	100.00
		keymgr_csr_rw	1.560s	114.687us	20	20	100.00
		keymgr_csr_aliasing	10.120s	762.402us	5	5	100.00
		keymgr_same_csr_outstanding	4.290s	119.830us	20	20	100.00
V2		TOTAL			731	740	98.78
V2S	sec_cm_additional_check	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
		keymgr_tl_intg_err	41.760s	8.998ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	16.140s	664.847us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	16.140s	664.847us	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	16.140s	664.847us	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	16.140s	664.847us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	14.670s	419.205us	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	41.760s	8.998ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	16.140s	664.847us	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	59.430s	2.294ms	49	50	98.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.158m	4.852ms	50	50	100.00
		keymgr_csr_rw	1.560s	114.687us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.158m	4.852ms	50	50	100.00
		keymgr_csr_rw	1.560s	114.687us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.158m	4.852ms	50	50	100.00
		keymgr_csr_rw	1.560s	114.687us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	7.610s	538.643us	50	50	100.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.359m	5.073ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.359m	5.073ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.158m	4.852ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	35.830s	4.328ms	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	1.272m	6.402ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	7.610s	538.643us	50	50	100.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	1.272m	6.402ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	1.272m	6.402ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	1.272m	6.402ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	26.960s	1.579ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	1.272m	6.402ms	49	50	98.00
V2S		TOTAL			164	165	99.39
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	28.240s	670.321us	13	50	26.00
V3		TOTAL			13	50	26.00
		TOTAL			1063	1110	95.77

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	13	81.25
V2S	6	6	5	83.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.80	99.07	97.99	98.39	100.00	99.11	98.41	91.63

Failure Buckets

UVM_ERROR (cip_base_vseq.sv:829) [keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 26 failures:
- Test keymgr_stress_all_with_rand_reset has 26 failures.
  - 1.keymgr_stress_all_with_rand_reset.66859922577823310486249794934799924850385453738679728850590780832834558483729
    Line 825, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/1.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 225189653 ps: (cip_base_vseq.sv:829) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 225189653 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 5.keymgr_stress_all_with_rand_reset.66944536022707096420651006391708523833568089432532202155388281652708321221554
    Line 457, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/5.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 144155279 ps: (cip_base_vseq.sv:829) [uvm_test_top.env.virtual_sequencer.keymgr_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 144155279 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 24 more failures.
UVM_ERROR (keymgr_scoreboard.sv:736) [scoreboard] Check failed item.d_data !=gmv(csr) (* [] vs * [])` has 11 failures:
- Test keymgr_stress_all_with_rand_reset has 11 failures.
  - 2.keymgr_stress_all_with_rand_reset.80380685219430656491968984614347008299769283629559310441900396978182483486152
    Line 1481, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 868909789 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (542346802 [0x20538e32] vs 542346802 [0x20538e32])
  UVM_INFO @ 868909789 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 10.keymgr_stress_all_with_rand_reset.71162910804628729277375549504544686434072515841377631741441849266655081257057
    Line 1747, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 665510899 ps: (keymgr_scoreboard.sv:736) [uvm_test_top.env.scoreboard] Check failed item.d_data != `gmv(csr) (854881907 [0x32f47673] vs 854881907 [0x32f47673])
  UVM_INFO @ 665510899 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 9 more failures.

UVM_ERROR (keymgr_scoreboard.sv:1013) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 7 failures:

Test keymgr_kmac_rsp_err has 7 failures.

2.keymgr_kmac_rsp_err.24239509432949267096637606487743856697644139804337031360927492816578868110686
Line 553, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/2.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 231986686 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (317040095251520922186931445469508330224640438428511604364475673402326356770608047143713373326625404149094175960730306478325507052983054454140946562866232579559204655311853042925100317905231056297770236228354523249282463274976728719880871127636548927779338029693134456292247208454876243609602472385021473764770712256796506378172208917868817657196890999746600243393899951579979452323163038211573926529827888 [0xd35e4d0000000000000000000000000000000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9dada5ca05746391ebcc006b7a37259adab4468a8cb96926bdc7bc99d230a0ac129d65ad4e576c6a7ece15b7e59eef4d009e5ab1194695c99bf201f1642515227cfe59278c4686c191dd92a0f8afb6a8126409f79376c47356b5283d8ec9b0f737d1294091efe20c65e8db2c19a438430] vs 317040095251520922186931445469508330224640438428511604364475673402326356770608047143713373326625404149094175960730306478325507052983054454140946562866232579559204655311853042925100317905231056297770236228354523249282463274976728719880871127636548927779338029693134456292247208454876243609602472385021473764770712256796506378172208917868817657196890999746600243393899951579979452323163038211573926529827888 [0xd35e4d0000000000000000000000000000000000000000003a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9dada5ca05746391ebcc006b7a37259adab4468a8cb96926bdc7bc99d230a0ac129d65ad4e576c6a7ece15b7e59eef4d009e5ab1194695c99bf201f1642515227cfe59278c4686c191dd92a0f8afb6a8126409f79376c47356b5283d8ec9b0f737d1294091efe20c65e8db2c19a438430]) cdi_type: Attestation
  
   DiversificationKey act: 0x26409f79376c47356b5283d8ec9b0f737d1294091efe20c65e8db2c19a438430, exp: 0x26409f79376c47356b5283d8ec9b0f737d1294091efe20c65e8db2c19a438430
   RomDigest act: 0x9e5ab1194695c99bf201f1642515227cfe59278c4686c191dd92a0f8afb6a81, exp: 0x9e5ab1194695c99bf201f1642515227cfe59278c4686c191dd92a0f8afb6a81
   HealthMeasurement act: 0x29d65ad4e576c6a7ece15b7e59eef4d0, exp: 0x29d65ad4e576c6a7ece15b7e59eef4d0

19.keymgr_kmac_rsp_err.49337438735060275508277907556041100674340492656095824951889790177719687144650
Line 459, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/19.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 190834633 ps: (keymgr_scoreboard.sv:1013) [uvm_test_top.env.scoreboard] Check failed act != exp (354432721907123137654888453992412283229793118397903359271410897924249427035625792647232313655166843547865663470524287517708317929436527237467748788944617519953862680881764682385740223315895892368855670457187610719304119276243814117321411934208312095348504667977438143360246993664998151327448328337018372262879374396451112437266736167404296506233050017326730466521577357302646964927807111805951270575499721 [0xec4c39ef0000000046800c76f120bdce00000000b67d1e383a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f93d20d58d6803e580fc8d718e2e90c20129bea8e15f2ea590776614df795f985edbab7166eec5913bc17c51cb1f2a87ff1d6960e43575acbb49fc650322133832d74629525ca89d639f4d824008087e135741b73a9436df24dbfd2d6e7ecc1c1d8905d9f6f1d8eba722e5179c92a969c9] vs 354432721907123137654888453992412283229793118397903359271410897924249427035625792647232313655166843547865663470524287517708317929436527237467748788944617519953862680881764682385740223315895892368855670457187610719304119276243814117321411934208312095348504667977438143360246993664998151327448328337018372262879374396451112437266736167404296506233050017326730466521577357302646964927807111805951270575499721 [0xec4c39ef0000000046800c76f120bdce00000000b67d1e383a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f93d20d58d6803e580fc8d718e2e90c20129bea8e15f2ea590776614df795f985edbab7166eec5913bc17c51cb1f2a87ff1d6960e43575acbb49fc650322133832d74629525ca89d639f4d824008087e135741b73a9436df24dbfd2d6e7ecc1c1d8905d9f6f1d8eba722e5179c92a969c9]) cdi_type: Attestation
  
   DiversificationKey act: 0x5741b73a9436df24dbfd2d6e7ecc1c1d8905d9f6f1d8eba722e5179c92a969c9, exp: 0x5741b73a9436df24dbfd2d6e7ecc1c1d8905d9f6f1d8eba722e5179c92a969c9
   RomDigest act: 0x1d6960e43575acbb49fc650322133832d74629525ca89d639f4d824008087e13, exp: 0x1d6960e43575acbb49fc650322133832d74629525ca89d639f4d824008087e13
   HealthMeasurement act: 0xdbab7166eec5913bc17c51cb1f2a87ff, exp: 0xdbab7166eec5913bc17c51cb1f2a87ff

... and 5 more failures.

UVM_ERROR (cip_base_scoreboard.sv:301) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 2 failures:
- Test keymgr_sw_invalid_input has 1 failures.
  - 10.keymgr_sw_invalid_input.99925841308677817653012433173254545302768630850061133425905172960251076928557
    Line 650, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/10.keymgr_sw_invalid_input/latest/run.log
```
  UVM_ERROR @ 160411714 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @ 160411714 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test keymgr_cfg_regwen has 1 failures.
  - 43.keymgr_cfg_regwen.53818082475403256828435083435074645523969023180941397245883309905932144105810
    Line 439, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/43.keymgr_cfg_regwen/latest/run.log
```
  UVM_ERROR @  23891629 ps: (cip_base_scoreboard.sv:301) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  23891629 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (keymgr_custom_cm_vseq.sv:81) [keymgr_custom_cm_vseq] wait timeout occurred! has 1 failures:
- Test keymgr_custom_cm has 1 failures.
  - 26.keymgr_custom_cm.44862649393069133465272268285251976048715020192653324257018996528209116893950
    Line 456, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/26.keymgr_custom_cm/latest/run.log
```
  UVM_FATAL @ 10789857225 ps: (keymgr_custom_cm_vseq.sv:81) [uvm_test_top.env.virtual_sequencer.keymgr_custom_cm_vseq] wait timeout occurred!
  UVM_INFO @ 10789857225 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```