AES/UNMASKED Simulation Results

Monday August 12 2024 23:02:30 UTC

GitHub Revision: `c082b8981f`

Branch: os_regression

Simulator: XCELIUM

Build randomization enabled with --build-seed 107262934208806092150901079363789224644653433402469901409990667510497383888850

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	wake_up	aes_wake_up	2.000s	58.818us	1	1	100.00
V1	smoke	aes_smoke	10.000s	436.407us	50	50	100.00
V1	csr_hw_reset	aes_csr_hw_reset	3.000s	58.624us	5	5	100.00
V1	csr_rw	aes_csr_rw	3.000s	55.678us	20	20	100.00
V1	csr_bit_bash	aes_csr_bit_bash	10.000s	1.235ms	5	5	100.00
V1	csr_aliasing	aes_csr_aliasing	4.000s	272.546us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	aes_csr_mem_rw_with_rand_reset	3.000s	162.520us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	aes_csr_rw	3.000s	55.678us	20	20	100.00
		aes_csr_aliasing	4.000s	272.546us	5	5	100.00
V1		TOTAL			106	106	100.00
V2	algorithm	aes_smoke	10.000s	436.407us	50	50	100.00
		aes_config_error	14.000s	85.929us	50	50	100.00
		aes_stress	9.000s	98.180us	50	50	100.00
V2	key_length	aes_smoke	10.000s	436.407us	50	50	100.00
		aes_config_error	14.000s	85.929us	50	50	100.00
		aes_stress	9.000s	98.180us	50	50	100.00
V2	back2back	aes_stress	9.000s	98.180us	50	50	100.00
		aes_b2b	10.000s	407.567us	50	50	100.00
V2	backpressure	aes_stress	9.000s	98.180us	50	50	100.00
V2	multi_message	aes_smoke	10.000s	436.407us	50	50	100.00
		aes_config_error	14.000s	85.929us	50	50	100.00
		aes_stress	9.000s	98.180us	50	50	100.00
		aes_alert_reset	9.000s	91.151us	48	50	96.00
V2	failure_test	aes_man_cfg_err	13.000s	84.277us	50	50	100.00
		aes_config_error	14.000s	85.929us	50	50	100.00
		aes_alert_reset	9.000s	91.151us	48	50	96.00
V2	trigger_clear_test	aes_clear	17.000s	321.926us	50	50	100.00
V2	nist_test_vectors	aes_nist_vectors	5.000s	169.830us	1	1	100.00
V2	reset_recovery	aes_alert_reset	9.000s	91.151us	48	50	96.00
V2	stress	aes_stress	9.000s	98.180us	50	50	100.00
V2	sideload	aes_stress	9.000s	98.180us	50	50	100.00
		aes_sideload	14.000s	63.598us	50	50	100.00
V2	deinitialization	aes_deinit	9.000s	65.506us	50	50	100.00
V2	stress_all	aes_stress_all	29.000s	3.037ms	10	10	100.00
V2	alert_test	aes_alert_test	12.000s	63.949us	50	50	100.00
V2	tl_d_oob_addr_access	aes_tl_errors	6.000s	378.493us	20	20	100.00
V2	tl_d_illegal_access	aes_tl_errors	6.000s	378.493us	20	20	100.00
V2	tl_d_outstanding_access	aes_csr_hw_reset	3.000s	58.624us	5	5	100.00
		aes_csr_rw	3.000s	55.678us	20	20	100.00
		aes_csr_aliasing	4.000s	272.546us	5	5	100.00
		aes_same_csr_outstanding	4.000s	61.794us	20	20	100.00
V2	tl_d_partial_access	aes_csr_hw_reset	3.000s	58.624us	5	5	100.00
		aes_csr_rw	3.000s	55.678us	20	20	100.00
		aes_csr_aliasing	4.000s	272.546us	5	5	100.00
		aes_same_csr_outstanding	4.000s	61.794us	20	20	100.00
V2		TOTAL			499	501	99.60
V2S	reseeding	aes_reseed	13.000s	119.718us	50	50	100.00
V2S	fault_inject	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
V2S	shadow_reg_update_error	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	shadow_reg_storage_error	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	shadowed_reset_glitch	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	aes_shadow_reg_errors_with_csr_rw	5.000s	1.590ms	20	20	100.00
V2S	tl_intg_err	aes_sec_cm	9.000s	826.166us	5	5	100.00
		aes_tl_intg_err	5.000s	258.693us	20	20	100.00
V2S	sec_cm_bus_integrity	aes_tl_intg_err	5.000s	258.693us	20	20	100.00
V2S	sec_cm_lc_escalate_en_intersig_mubi	aes_alert_reset	9.000s	91.151us	48	50	96.00
V2S	sec_cm_main_config_shadow	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	sec_cm_main_config_sparse	aes_smoke	10.000s	436.407us	50	50	100.00
		aes_stress	9.000s	98.180us	50	50	100.00
		aes_alert_reset	9.000s	91.151us	48	50	96.00
		aes_core_fi	2.367m	10.030ms	64	70	91.43
V2S	sec_cm_aux_config_shadow	aes_shadow_reg_errors	3.000s	133.175us	20	20	100.00
V2S	sec_cm_aux_config_regwen	aes_readability	8.000s	175.317us	50	50	100.00
		aes_stress	9.000s	98.180us	50	50	100.00
V2S	sec_cm_key_sideload	aes_stress	9.000s	98.180us	50	50	100.00
		aes_sideload	14.000s	63.598us	50	50	100.00
V2S	sec_cm_key_sw_unreadable	aes_readability	8.000s	175.317us	50	50	100.00
V2S	sec_cm_data_reg_sw_unreadable	aes_readability	8.000s	175.317us	50	50	100.00
V2S	sec_cm_key_sec_wipe	aes_readability	8.000s	175.317us	50	50	100.00
V2S	sec_cm_iv_config_sec_wipe	aes_readability	8.000s	175.317us	50	50	100.00
V2S	sec_cm_data_reg_sec_wipe	aes_readability	8.000s	175.317us	50	50	100.00
V2S	sec_cm_data_reg_key_sca	aes_stress	9.000s	98.180us	50	50	100.00
V2S	sec_cm_key_masking	aes_stress	9.000s	98.180us	50	50	100.00
V2S	sec_cm_main_fsm_sparse	aes_fi	12.000s	44.229us	46	50	92.00
V2S	sec_cm_main_fsm_redun	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_cipher_fsm_sparse	aes_fi	12.000s	44.229us	46	50	92.00
V2S	sec_cm_cipher_fsm_redun	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
V2S	sec_cm_cipher_ctr_redun	aes_cipher_fi	52.000s	65.605ms	322	350	92.00
V2S	sec_cm_ctr_fsm_sparse	aes_fi	12.000s	44.229us	46	50	92.00
V2S	sec_cm_ctr_fsm_redun	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_ctrl_sparse	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_main_fsm_global_esc	aes_alert_reset	9.000s	91.151us	48	50	96.00
V2S	sec_cm_main_fsm_local_esc	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_cipher_fsm_local_esc	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_ctr_fsm_local_esc	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_ctr_fi	13.000s	55.553us	50	50	100.00
V2S	sec_cm_data_reg_local_esc	aes_fi	12.000s	44.229us	46	50	92.00
		aes_control_fi	43.000s	10.003ms	278	300	92.67
		aes_cipher_fi	52.000s	65.605ms	322	350	92.00
V2S		TOTAL			925	985	93.91
V3	stress_all_with_rand_reset	aes_stress_all_with_rand_reset	28.000s	369.735us	0	10	0.00
V3		TOTAL			0	10	0.00
		TOTAL			1530	1602	95.51

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	13	13	12	92.31
V2S	11	11	7	63.64
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

Score	Block	Branch	Statement	Expression	Toggle	Fsm	Assertion	CoverGroup
97.21	97.55	94.48	98.77	93.80	97.72	91.11	98.66	96.21

Failure Buckets

Job aes_unmasked-sim-xcelium_run_default killed due to: Exit reason: User job exceeded runlimit: User job timed out has 33 failures:
- Test aes_control_fi has 15 failures.
  - 13.aes_control_fi.101420042947431780017111279005712007080358323083236538546912951161765127787465
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/13.aes_control_fi/latest/run.log
```
  Job ID: smart:3318dfd3-5198-43ff-aef6-72afd5a8ba68
```
  - 66.aes_control_fi.83543056334244505642119890766226613694054682335263858286282253929090465452585
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/66.aes_control_fi/latest/run.log
```
  Job ID: smart:6e5bc5e5-1125-46ad-8725-da53807a8b7c
```
  - ... and 13 more failures.
- Test aes_cipher_fi has 18 failures.
  - 22.aes_cipher_fi.79814703816582407587190230257940433734515985706602647762703176280668879099058
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/22.aes_cipher_fi/latest/run.log
```
  Job ID: smart:fca7921f-f2c3-4ed3-b373-017ca24c8018
```
  - 23.aes_cipher_fi.83619893119468379832990726669687234103998274693849593934667055916867821122650
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/23.aes_cipher_fi/latest/run.log
```
  Job ID: smart:15b74d67-a19b-458a-8b73-f33b68485425
```
  - ... and 16 more failures.
UVM_FATAL (aes_cipher_fi_vseq.sv:62) [aes_cipher_fi_vseq] wait timeout occurred! has 10 failures:
- Test aes_cipher_fi has 10 failures.
  - 27.aes_cipher_fi.54928191613280896461226692225163581607460358402854630029224678970332770813656
    Line 312, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/27.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10017955670 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10017955670 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 49.aes_cipher_fi.30521631998537296154076119548834363450870971386430513359610665667649446479997
    Line 317, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/49.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10004741341 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10004741341 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 8 more failures.
UVM_FATAL (aes_control_fi_vseq.sv:62) [aes_control_fi_vseq] wait timeout occurred! has 7 failures:
- Test aes_control_fi has 7 failures.
  - 16.aes_control_fi.88241420115284810710789382024377514774946734347684765696496845651698208149709
    Line 324, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/16.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10018827956 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10018827956 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 51.aes_control_fi.84785441101319015435563519211399068078066754048257203782085401439054771400378
    Line 321, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/51.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10004775036 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10004775036 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 5 more failures.

UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 4 failures:

Test aes_stress_all_with_rand_reset has 4 failures.

4.aes_stress_all_with_rand_reset.38413111525905564417378691739619627077473132568812519589556299162805012533454
Line 1553, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/4.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 798421119 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 798421119 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

5.aes_stress_all_with_rand_reset.112552713980787613202399022047064721478610718696832949164105651403232433945967
Line 1529, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/5.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 354008410 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 354008410 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 2 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_*/rtl/aes_core.sv,993): Assertion AesSecCmDataRegLocalEscIv has failed (* cycles, starting * PS) has 3 failures:

Test aes_fi has 2 failures.

0.aes_fi.87909470176355431104601339351223145522318643467735167819232986430724657408443
Line 877, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/0.aes_fi/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,993): (time 16631104 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscIv has failed (2 cycles, starting 16620687 PS)
  UVM_ERROR @  16631104 ps: (aes_core.sv:993) [ASSERT FAILED] AesSecCmDataRegLocalEscIv
  UVM_INFO @  16631104 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

48.aes_fi.44755481884940392060532017533468123999685272174163725138652014029166020965966
Line 2596, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/48.aes_fi/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,993): (time 44228818 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscIv has failed (2 cycles, starting 44166318 PS)
  UVM_ERROR @  44228818 ps: (aes_core.sv:993) [ASSERT FAILED] AesSecCmDataRegLocalEscIv
  UVM_INFO @  44228818 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test aes_alert_reset has 1 failures.

3.aes_alert_reset.76773927879800796139002657469194551081618748879650590174020533603153298116291
Line 3038, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/3.aes_alert_reset/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,993): (time 8356111 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscIv has failed (2 cycles, starting 8345802 PS)
  UVM_ERROR @   8356111 ps: (aes_core.sv:993) [ASSERT FAILED] AesSecCmDataRegLocalEscIv
  UVM_INFO @   8356111 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_FATAL (aes_base_vseq.sv:74) [aes_alert_reset_vseq] Check failed (aes_ctrl_aux[*] == cfg.do_reseed) has 3 failures:
- Test aes_stress_all_with_rand_reset has 3 failures.
  - 0.aes_stress_all_with_rand_reset.23463989719785955701626876262741843226509459218244168947291600039382432548193
    Line 345, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/0.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @  32360425 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_alert_reset_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @  32360425 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 2.aes_stress_all_with_rand_reset.27824148620201122321339719090170019448293375129708545713932293241047481045833
    Line 540, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/2.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @ 223624024 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_alert_reset_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @ 223624024 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_*/rtl/aes_core.sv,987): Assertion AesSecCmDataRegLocalEscDataOut has failed (* cycles, starting * PS) has 3 failures:

Test aes_stress_all_with_rand_reset has 1 failures.

1.aes_stress_all_with_rand_reset.115483723893047812931168002082409885340421821319153232333821668331909852181109
Line 360, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/1.aes_stress_all_with_rand_reset/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 147006237 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 146953605 PS)
        ($past(iv_q) != $past(state_done_transposed, 2) ^ $past(data_in_prev_q, 2)))
                                                                                   |
  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,993): (time 147006237 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscIv has failed (2 cycles, starting 146953605 PS)
  UVM_ERROR @ 147006237 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut

Test aes_alert_reset has 1 failures.

8.aes_alert_reset.98130733606902384589876778921684135124926564536234872193003377477992644859823
Line 2883, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/8.aes_alert_reset/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 8203135 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 8193135 PS)
  UVM_ERROR @   8203135 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut
  UVM_INFO @   8203135 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test aes_fi has 1 failures.

35.aes_fi.61420311918983575127450108264624551048144334249054519507869304276218195153618
Line 2559, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/35.aes_fi/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 35382184 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 35329552 PS)
  UVM_ERROR @  35382184 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut
  UVM_INFO @  35382184 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_FATAL (aes_core_fi_vseq.sv:89) [aes_core_fi_vseq] wait timeout occurred! has 3 failures:
- Test aes_core_fi has 3 failures.
  - 21.aes_core_fi.29148269061269706801350084554847649083391691236356216338906161349009276944354
    Line 314, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/21.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10059856391 ps: (aes_core_fi_vseq.sv:89) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10059856391 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 24.aes_core_fi.68731031861178493312438358737167574885813548136087879044703016392863806306059
    Line 315, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/24.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10036644074 ps: (aes_core_fi_vseq.sv:89) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10036644074 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.
Exit reason: Error: User command failed UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 1 failures:
- Test aes_stress_all_with_rand_reset has 1 failures.
  - 3.aes_stress_all_with_rand_reset.107869592821297179310251689030633212480595187356343881544834261107955650566289
    Line 1309, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/3.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 383543079 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_stress_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 383543079 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_core_fi_vseq.sv:66) [aes_core_fi_vseq] wait timeout occurred! has 1 failures:
- Test aes_core_fi has 1 failures.
  - 7.aes_core_fi.17362341489274718115259909862934481166122433157425777154291202893645578387769
    Line 316, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/7.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10007113615 ps: (aes_core_fi_vseq.sv:66) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10007113615 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_base_vseq.sv:74) [aes_reseed_vseq] Check failed (aes_ctrl_aux[*] == cfg.do_reseed) has 1 failures:
- Test aes_stress_all_with_rand_reset has 1 failures.
  - 7.aes_stress_all_with_rand_reset.92545870165189232419400529793857194171023949789693279538379510630432146006769
    Line 326, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/7.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @  89513164 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_reseed_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @  89513164 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_fi_vseq.sv:69) virtual_sequencer [aes_fi_vseq] Was Able to finish without clearing reset has 1 failures:
- Test aes_fi has 1 failures.
  - 9.aes_fi.40176582057092084724624629618937245901501503036198919150685570936628519770877
    Line 2647, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/9.aes_fi/latest/run.log
```
  UVM_FATAL @  12378287 ps: (aes_fi_vseq.sv:69) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.aes_fi_vseq] Was Able to finish without clearing reset
  UVM_INFO @  12378287 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=*, Comparison=CompareOpEq, exp_data=*, call_count=10) has 1 failures:
- Test aes_core_fi has 1 failures.
  - 39.aes_core_fi.103278223772731140031983377160956114138191918288002337306674528327805824889347
    Line 315, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/39.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10029687173 ps: (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=0xf270a684, Comparison=CompareOpEq, exp_data=0x0, call_count=10)
  UVM_INFO @ 10029687173 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=*, Comparison=CompareOpEq, exp_data=*, call_count=9) has 1 failures:
- Test aes_core_fi has 1 failures.
  - 56.aes_core_fi.617194350188992342521950318583907178689525405504840334613933688116311459167
    Line 318, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/56.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10147955663 ps: (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=0x5c360284, Comparison=CompareOpEq, exp_data=0x0, call_count=9)
  UVM_INFO @ 10147955663 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```