AES/MASKED Simulation Results

Thursday August 15 2024 23:02:21 UTC

GitHub Revision: `d09e282b26`

Branch: os_regression

Simulator: XCELIUM

Build randomization enabled with --build-seed 47908880312501934977153450267828796412449789719488445881682136509150457490963

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	wake_up	aes_wake_up	4.000s	63.688us	1	1	100.00
V1	smoke	aes_smoke	10.000s	289.923us	50	50	100.00
V1	csr_hw_reset	aes_csr_hw_reset	3.000s	182.549us	5	5	100.00
V1	csr_rw	aes_csr_rw	3.000s	137.992us	20	20	100.00
V1	csr_bit_bash	aes_csr_bit_bash	9.000s	1.037ms	5	5	100.00
V1	csr_aliasing	aes_csr_aliasing	7.000s	1.401ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	aes_csr_mem_rw_with_rand_reset	4.000s	323.062us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	aes_csr_rw	3.000s	137.992us	20	20	100.00
		aes_csr_aliasing	7.000s	1.401ms	5	5	100.00
V1		TOTAL			106	106	100.00
V2	algorithm	aes_smoke	10.000s	289.923us	50	50	100.00
		aes_config_error	14.000s	330.609us	50	50	100.00
		aes_stress	58.000s	4.447ms	50	50	100.00
V2	key_length	aes_smoke	10.000s	289.923us	50	50	100.00
		aes_config_error	14.000s	330.609us	50	50	100.00
		aes_stress	58.000s	4.447ms	50	50	100.00
V2	back2back	aes_stress	58.000s	4.447ms	50	50	100.00
		aes_b2b	1.000m	677.847us	50	50	100.00
V2	backpressure	aes_stress	58.000s	4.447ms	50	50	100.00
V2	multi_message	aes_smoke	10.000s	289.923us	50	50	100.00
		aes_config_error	14.000s	330.609us	50	50	100.00
		aes_stress	58.000s	4.447ms	50	50	100.00
		aes_alert_reset	2.200m	5.829ms	50	50	100.00
V2	failure_test	aes_man_cfg_err	9.000s	65.436us	50	50	100.00
		aes_config_error	14.000s	330.609us	50	50	100.00
		aes_alert_reset	2.200m	5.829ms	50	50	100.00
V2	trigger_clear_test	aes_clear	2.767m	5.204ms	50	50	100.00
V2	nist_test_vectors	aes_nist_vectors	15.000s	691.475us	1	1	100.00
V2	reset_recovery	aes_alert_reset	2.200m	5.829ms	50	50	100.00
V2	stress	aes_stress	58.000s	4.447ms	50	50	100.00
V2	sideload	aes_stress	58.000s	4.447ms	50	50	100.00
		aes_sideload	38.000s	2.003ms	50	50	100.00
V2	deinitialization	aes_deinit	12.000s	323.012us	50	50	100.00
V2	stress_all	aes_stress_all	1.683m	2.775ms	9	10	90.00
V2	alert_test	aes_alert_test	14.000s	57.841us	50	50	100.00
V2	tl_d_oob_addr_access	aes_tl_errors	6.000s	1.370ms	20	20	100.00
V2	tl_d_illegal_access	aes_tl_errors	6.000s	1.370ms	20	20	100.00
V2	tl_d_outstanding_access	aes_csr_hw_reset	3.000s	182.549us	5	5	100.00
		aes_csr_rw	3.000s	137.992us	20	20	100.00
		aes_csr_aliasing	7.000s	1.401ms	5	5	100.00
		aes_same_csr_outstanding	4.000s	201.265us	20	20	100.00
V2	tl_d_partial_access	aes_csr_hw_reset	3.000s	182.549us	5	5	100.00
		aes_csr_rw	3.000s	137.992us	20	20	100.00
		aes_csr_aliasing	7.000s	1.401ms	5	5	100.00
		aes_same_csr_outstanding	4.000s	201.265us	20	20	100.00
V2		TOTAL			500	501	99.80
V2S	reseeding	aes_reseed	38.000s	1.105ms	50	50	100.00
V2S	fault_inject	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
V2S	shadow_reg_update_error	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	shadow_reg_storage_error	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	shadowed_reset_glitch	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	aes_shadow_reg_errors_with_csr_rw	4.000s	90.165us	20	20	100.00
V2S	tl_intg_err	aes_sec_cm	15.000s	1.350ms	5	5	100.00
		aes_tl_intg_err	6.000s	770.251us	20	20	100.00
V2S	sec_cm_bus_integrity	aes_tl_intg_err	6.000s	770.251us	20	20	100.00
V2S	sec_cm_lc_escalate_en_intersig_mubi	aes_alert_reset	2.200m	5.829ms	50	50	100.00
V2S	sec_cm_main_config_shadow	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	sec_cm_main_config_sparse	aes_smoke	10.000s	289.923us	50	50	100.00
		aes_stress	58.000s	4.447ms	50	50	100.00
		aes_alert_reset	2.200m	5.829ms	50	50	100.00
		aes_core_fi	1.400m	10.003ms	67	70	95.71
V2S	sec_cm_aux_config_shadow	aes_shadow_reg_errors	4.000s	61.192us	20	20	100.00
V2S	sec_cm_aux_config_regwen	aes_readability	9.000s	96.974us	50	50	100.00
		aes_stress	58.000s	4.447ms	50	50	100.00
V2S	sec_cm_key_sideload	aes_stress	58.000s	4.447ms	50	50	100.00
		aes_sideload	38.000s	2.003ms	50	50	100.00
V2S	sec_cm_key_sw_unreadable	aes_readability	9.000s	96.974us	50	50	100.00
V2S	sec_cm_data_reg_sw_unreadable	aes_readability	9.000s	96.974us	50	50	100.00
V2S	sec_cm_key_sec_wipe	aes_readability	9.000s	96.974us	50	50	100.00
V2S	sec_cm_iv_config_sec_wipe	aes_readability	9.000s	96.974us	50	50	100.00
V2S	sec_cm_data_reg_sec_wipe	aes_readability	9.000s	96.974us	50	50	100.00
V2S	sec_cm_data_reg_key_sca	aes_stress	58.000s	4.447ms	50	50	100.00
V2S	sec_cm_key_masking	aes_stress	58.000s	4.447ms	50	50	100.00
V2S	sec_cm_main_fsm_sparse	aes_fi	41.000s	1.548ms	49	50	98.00
V2S	sec_cm_main_fsm_redun	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_cipher_fsm_sparse	aes_fi	41.000s	1.548ms	49	50	98.00
V2S	sec_cm_cipher_fsm_redun	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
V2S	sec_cm_cipher_ctr_redun	aes_cipher_fi	40.000s	10.005ms	342	350	97.71
V2S	sec_cm_ctr_fsm_sparse	aes_fi	41.000s	1.548ms	49	50	98.00
V2S	sec_cm_ctr_fsm_redun	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_ctrl_sparse	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_main_fsm_global_esc	aes_alert_reset	2.200m	5.829ms	50	50	100.00
V2S	sec_cm_main_fsm_local_esc	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_cipher_fsm_local_esc	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_ctr_fsm_local_esc	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_ctr_fi	5.000s	97.326us	50	50	100.00
V2S	sec_cm_data_reg_local_esc	aes_fi	41.000s	1.548ms	49	50	98.00
		aes_control_fi	49.000s	10.012ms	284	300	94.67
		aes_cipher_fi	40.000s	10.005ms	342	350	97.71
V2S		TOTAL			957	985	97.16
V3	stress_all_with_rand_reset	aes_stress_all_with_rand_reset	42.000s	1.141ms	0	10	0.00
V3		TOTAL			0	10	0.00
		TOTAL			1563	1602	97.57

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	13	13	12	92.31
V2S	11	11	7	63.64
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

Score	Block	Branch	Statement	Expression	Toggle	Fsm	Assertion	CoverGroup
98.32	98.54	96.28	99.43	95.67	97.72	97.78	98.96	96.21

Failure Buckets

Job aes_masked-sim-xcelium_run_default killed due to: Exit reason: User job exceeded runlimit: User job timed out has 15 failures:
- Test aes_control_fi has 12 failures.
  - 26.aes_control_fi.84914735956886894651385125293128208764058051992147280969862319896174955005578
    Log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/26.aes_control_fi/latest/run.log
```
  Job ID: smart:d6dfffff-1bc8-4c30-a036-1986e59253cf
```
  - 40.aes_control_fi.10249232545607961535612799140902159992851133221692692614731962403830505480470
    Log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/40.aes_control_fi/latest/run.log
```
  Job ID: smart:cb72752c-4721-4488-a652-f35a798d7e1c
```
  - ... and 10 more failures.
- Test aes_cipher_fi has 3 failures.
  - 48.aes_cipher_fi.115731057290047147467577238292000047534544167068377454010294819111635554857096
    Log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/48.aes_cipher_fi/latest/run.log
```
  Job ID: smart:347d8bee-4527-4a32-8c71-0aed6090bd55
```
  - 121.aes_cipher_fi.16074559087378682892171118333739772855138286090699730042153664396498438400320
    Log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/121.aes_cipher_fi/latest/run.log
```
  Job ID: smart:8855451a-cf87-4e9b-8c4d-d1cbbd08fd26
```
  - ... and 1 more failures.
UVM_FATAL (aes_cipher_fi_vseq.sv:62) [aes_cipher_fi_vseq] wait timeout occurred! has 5 failures:
- Test aes_cipher_fi has 5 failures.
  - 27.aes_cipher_fi.97595067403978954132330038358413444237305886097443759680193620741596166169448
    Line 312, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/27.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10014233177 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10014233177 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 66.aes_cipher_fi.7050570374445988195276475345329486826227927252206326356944883352883516712077
    Line 324, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/66.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10009332721 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10009332721 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 3 more failures.
UVM_FATAL (aes_control_fi_vseq.sv:62) [aes_control_fi_vseq] wait timeout occurred! has 4 failures:
- Test aes_control_fi has 4 failures.
  - 66.aes_control_fi.553856409796359240500385924946305932285176221335815598994082172666789345915
    Line 322, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/66.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10017640022 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10017640022 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 137.aes_control_fi.73024752583876594608973501520363314570376126395590423974437174562120115632957
    Line 322, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/137.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10024085675 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10024085675 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 2 more failures.

UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 3 failures:

Test aes_stress_all_with_rand_reset has 3 failures.

2.aes_stress_all_with_rand_reset.34702271775864516312848086224905092807732821188789485489198101974004039285820
Line 403, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/2.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 7317668202 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 7317668202 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

5.aes_stress_all_with_rand_reset.75437495544328808440778000979108252497440146957492917414535349219407655848728
Line 604, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/5.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 731026546 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 731026546 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

UVM_FATAL (aes_core_fi_vseq.sv:66) [aes_core_fi_vseq] wait timeout occurred! has 3 failures:
- Test aes_core_fi has 3 failures.
  - 26.aes_core_fi.33315538600474597300239277493874636441047585955556447539968555753307993350060
    Line 317, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/26.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10014252536 ps: (aes_core_fi_vseq.sv:66) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10014252536 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 33.aes_core_fi.3249627691022654127106955113380080956563350487942908725124225140522512693730
    Line 321, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/33.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10012193388 ps: (aes_core_fi_vseq.sv:66) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10012193388 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.
Exit reason: Error: User command failed UVM_ERROR (cip_base_vseq.sv:868) [aes_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 2 failures:
- Test aes_stress_all_with_rand_reset has 2 failures.
  - 1.aes_stress_all_with_rand_reset.97687683859903746298720751969087009466812635186936254570421274270517054925043
    Line 490, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/1.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 741408383 ps: (cip_base_vseq.sv:868) [uvm_test_top.env.virtual_sequencer.aes_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 741408383 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 4.aes_stress_all_with_rand_reset.26108568826570071664730774100392324918271727061578802381767200274992718880318
    Line 563, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/4.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 475748840 ps: (cip_base_vseq.sv:868) [uvm_test_top.env.virtual_sequencer.aes_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 475748840 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_base_vseq.sv:74) [aes_alert_reset_vseq] Check failed (aes_ctrl_aux[*] == cfg.do_reseed) has 2 failures:
- Test aes_stress_all_with_rand_reset has 2 failures.
  - 3.aes_stress_all_with_rand_reset.24478158991447772729401703067334989765842163357874096711414786039017419766550
    Line 350, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/3.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @  37023603 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_alert_reset_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @  37023603 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 8.aes_stress_all_with_rand_reset.51983226028172639019952687154118037882889791853164859616690055567657704166186
    Line 343, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/8.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @  94046055 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_alert_reset_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @  94046055 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_base_vseq.sv:74) [aes_stress_vseq] Check failed (aes_ctrl_aux[*] == cfg.do_reseed) has 2 failures:
- Test aes_stress_all_with_rand_reset has 2 failures.
  - 6.aes_stress_all_with_rand_reset.81700471014134851052398770917361948017073843184380550075445937791518163634974
    Line 1314, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/6.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @ 5535075831 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_stress_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @ 5535075831 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 9.aes_stress_all_with_rand_reset.35980518487095453821219511072790698139574265333307288205505912852851036568467
    Line 323, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/9.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_FATAL @  59361703 ps: (aes_base_vseq.sv:74) [uvm_test_top.env.virtual_sequencer.aes_stress_vseq] Check failed (aes_ctrl_aux[0] == cfg.do_reseed)
  UVM_INFO @  59361703 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
Exit reason: Error: User command failed UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 1 failures:
- Test aes_stress_all_with_rand_reset has 1 failures.
  - 0.aes_stress_all_with_rand_reset.13525006433739167940873963539198688017535906823250032272549607389150878871019
    Line 1131, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/0.aes_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1140912887 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_stress_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 1140912887 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (cip_base_vseq.sv:557) [aes_alert_reset_vseq] Check failed * == cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() (* [*] vs * [*]) Alert fatal_fault fired unexpectedly! has 1 failures:
- Test aes_stress_all has 1 failures.
  - 2.aes_stress_all.113124521565049890673697167072982581962221203595876176041573663176377328705949
    Line 7166, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/2.aes_stress_all/latest/run.log
```
  UVM_ERROR @  61367205 ps: (cip_base_vseq.sv:557) [uvm_test_top.env.virtual_sequencer.aes_alert_reset_vseq] Check failed 0 == cfg.m_alert_agent_cfgs[alert_name].vif.get_alert() (0 [0x0] vs 1 [0x1]) Alert fatal_fault fired unexpectedly!
  UVM_INFO @  61367205 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_*/rtl/aes_core.sv,987): Assertion AesSecCmDataRegLocalEscDataOut has failed (* cycles, starting * PS) has 1 failures:

Test aes_fi has 1 failures.

9.aes_fi.100704988472934829817723993919683255144255757520250273668855648679599343347077
Line 1670, in log /container/opentitan-public/scratch/os_regression/aes_masked-sim-xcelium/9.aes_fi/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 5362445 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 5352445 PS)
        ($past(iv_q) != $past(state_done_transposed, 2) ^ $past(data_in_prev_q, 2)))
                                                                                   |
  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,993): (time 5362445 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscIv has failed (2 cycles, starting 5352445 PS)
  UVM_ERROR @   5362445 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut