AES/UNMASKED Simulation Results

Monday July 22 2024 23:02:17 UTC

GitHub Revision: `3e0219a2c5`

Branch: os_regression

Simulator: XCELIUM

Build randomization enabled with --build-seed 78193674045195286552709223969981662100934453993551616519215297815848091296886

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	wake_up	aes_wake_up	3.000s	91.472us	1	1	100.00
V1	smoke	aes_smoke	8.000s	130.130us	50	50	100.00
V1	csr_hw_reset	aes_csr_hw_reset	3.000s	72.633us	5	5	100.00
V1	csr_rw	aes_csr_rw	4.000s	171.568us	20	20	100.00
V1	csr_bit_bash	aes_csr_bit_bash	11.000s	1.619ms	5	5	100.00
V1	csr_aliasing	aes_csr_aliasing	5.000s	180.375us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	aes_csr_mem_rw_with_rand_reset	4.000s	398.951us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	aes_csr_rw	4.000s	171.568us	20	20	100.00
		aes_csr_aliasing	5.000s	180.375us	5	5	100.00
V1		TOTAL			106	106	100.00
V2	algorithm	aes_smoke	8.000s	130.130us	50	50	100.00
		aes_config_error	9.000s	272.615us	50	50	100.00
		aes_stress	30.000s	327.644us	50	50	100.00
V2	key_length	aes_smoke	8.000s	130.130us	50	50	100.00
		aes_config_error	9.000s	272.615us	50	50	100.00
		aes_stress	30.000s	327.644us	50	50	100.00
V2	back2back	aes_stress	30.000s	327.644us	50	50	100.00
		aes_b2b	35.000s	142.300us	50	50	100.00
V2	backpressure	aes_stress	30.000s	327.644us	50	50	100.00
V2	multi_message	aes_smoke	8.000s	130.130us	50	50	100.00
		aes_config_error	9.000s	272.615us	50	50	100.00
		aes_stress	30.000s	327.644us	50	50	100.00
		aes_alert_reset	15.000s	70.971us	49	50	98.00
V2	failure_test	aes_man_cfg_err	16.000s	86.791us	50	50	100.00
		aes_config_error	9.000s	272.615us	50	50	100.00
		aes_alert_reset	15.000s	70.971us	49	50	98.00
V2	trigger_clear_test	aes_clear	27.000s	91.142us	50	50	100.00
V2	nist_test_vectors	aes_nist_vectors	5.000s	309.649us	1	1	100.00
V2	reset_recovery	aes_alert_reset	15.000s	70.971us	49	50	98.00
V2	stress	aes_stress	30.000s	327.644us	50	50	100.00
V2	sideload	aes_stress	30.000s	327.644us	50	50	100.00
		aes_sideload	18.000s	161.697us	50	50	100.00
V2	deinitialization	aes_deinit	16.000s	133.201us	50	50	100.00
V2	stress_all	aes_stress_all	41.000s	2.120ms	9	10	90.00
V2	alert_test	aes_alert_test	12.000s	97.618us	50	50	100.00
V2	tl_d_oob_addr_access	aes_tl_errors	6.000s	947.988us	20	20	100.00
V2	tl_d_illegal_access	aes_tl_errors	6.000s	947.988us	20	20	100.00
V2	tl_d_outstanding_access	aes_csr_hw_reset	3.000s	72.633us	5	5	100.00
		aes_csr_rw	4.000s	171.568us	20	20	100.00
		aes_csr_aliasing	5.000s	180.375us	5	5	100.00
		aes_same_csr_outstanding	5.000s	259.609us	20	20	100.00
V2	tl_d_partial_access	aes_csr_hw_reset	3.000s	72.633us	5	5	100.00
		aes_csr_rw	4.000s	171.568us	20	20	100.00
		aes_csr_aliasing	5.000s	180.375us	5	5	100.00
		aes_same_csr_outstanding	5.000s	259.609us	20	20	100.00
V2		TOTAL			499	501	99.60
V2S	reseeding	aes_reseed	14.000s	293.562us	50	50	100.00
V2S	fault_inject	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
V2S	shadow_reg_update_error	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	shadow_reg_storage_error	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	shadowed_reset_glitch	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	aes_shadow_reg_errors_with_csr_rw	4.000s	104.785us	20	20	100.00
V2S	tl_intg_err	aes_sec_cm	7.000s	3.025ms	5	5	100.00
		aes_tl_intg_err	5.000s	613.642us	20	20	100.00
V2S	sec_cm_bus_integrity	aes_tl_intg_err	5.000s	613.642us	20	20	100.00
V2S	sec_cm_lc_escalate_en_intersig_mubi	aes_alert_reset	15.000s	70.971us	49	50	98.00
V2S	sec_cm_main_config_shadow	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	sec_cm_main_config_sparse	aes_smoke	8.000s	130.130us	50	50	100.00
		aes_stress	30.000s	327.644us	50	50	100.00
		aes_alert_reset	15.000s	70.971us	49	50	98.00
		aes_core_fi	6.867m	10.008ms	65	70	92.86
V2S	sec_cm_aux_config_shadow	aes_shadow_reg_errors	4.000s	95.794us	20	20	100.00
V2S	sec_cm_aux_config_regwen	aes_readability	27.000s	76.449us	50	50	100.00
		aes_stress	30.000s	327.644us	50	50	100.00
V2S	sec_cm_key_sideload	aes_stress	30.000s	327.644us	50	50	100.00
		aes_sideload	18.000s	161.697us	50	50	100.00
V2S	sec_cm_key_sw_unreadable	aes_readability	27.000s	76.449us	50	50	100.00
V2S	sec_cm_data_reg_sw_unreadable	aes_readability	27.000s	76.449us	50	50	100.00
V2S	sec_cm_key_sec_wipe	aes_readability	27.000s	76.449us	50	50	100.00
V2S	sec_cm_iv_config_sec_wipe	aes_readability	27.000s	76.449us	50	50	100.00
V2S	sec_cm_data_reg_sec_wipe	aes_readability	27.000s	76.449us	50	50	100.00
V2S	sec_cm_data_reg_key_sca	aes_stress	30.000s	327.644us	50	50	100.00
V2S	sec_cm_key_masking	aes_stress	30.000s	327.644us	50	50	100.00
V2S	sec_cm_main_fsm_sparse	aes_fi	14.000s	75.703us	49	50	98.00
V2S	sec_cm_main_fsm_redun	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_cipher_fsm_sparse	aes_fi	14.000s	75.703us	49	50	98.00
V2S	sec_cm_cipher_fsm_redun	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
V2S	sec_cm_cipher_ctr_redun	aes_cipher_fi	53.000s	200.000ms	324	350	92.57
V2S	sec_cm_ctr_fsm_sparse	aes_fi	14.000s	75.703us	49	50	98.00
V2S	sec_cm_ctr_fsm_redun	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_ctrl_sparse	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_main_fsm_global_esc	aes_alert_reset	15.000s	70.971us	49	50	98.00
V2S	sec_cm_main_fsm_local_esc	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_cipher_fsm_local_esc	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_ctr_fsm_local_esc	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_ctr_fi	13.000s	127.079us	49	50	98.00
V2S	sec_cm_data_reg_local_esc	aes_fi	14.000s	75.703us	49	50	98.00
		aes_control_fi	46.000s	20.757ms	264	300	88.00
		aes_cipher_fi	53.000s	200.000ms	324	350	92.57
V2S		TOTAL			916	985	92.99
V3	stress_all_with_rand_reset	aes_stress_all_with_rand_reset	9.600m	18.108ms	0	10	0.00
V3		TOTAL			0	10	0.00
		TOTAL			1521	1602	94.94

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	13	13	11	84.62
V2S	11	11	6	54.55
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

Score	Block	Branch	Statement	Expression	Toggle	Fsm	Assertion	CoverGroup
97.16	97.52	94.39	98.77	93.60	97.64	91.11	98.85	95.81

Failure Buckets

Job aes_unmasked-sim-xcelium_run_default killed due to: Exit reason: User job exceeded runlimit: User job timed out has 50 failures:
- Test aes_control_fi has 31 failures.
  - 12.aes_control_fi.104127250324175603504728289083521780277431144043479641440849304958400603343857
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/12.aes_control_fi/latest/run.log
```
  Job ID: smart:23b49584-660b-4cea-a308-cd2926676108
```
  - 23.aes_control_fi.5962877195429877985460299341949951662025911963785087196890145184312284626999
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/23.aes_control_fi/latest/run.log
```
  Job ID: smart:71d2f790-1af5-4a7a-b5a6-19b2e41db8da
```
  - ... and 29 more failures.
- Test aes_cipher_fi has 18 failures.
  - 15.aes_cipher_fi.97661740335175152386538921780725069969329053715363333684718870748650225535449
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/15.aes_cipher_fi/latest/run.log
```
  Job ID: smart:9c8b3f97-7abd-45a7-8392-a3f552b08bcb
```
  - 27.aes_cipher_fi.57736129563539515997123023928711119207430593107184514629200851271743504044436
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/27.aes_cipher_fi/latest/run.log
```
  Job ID: smart:e21b62b4-df91-4c9b-a3e2-24ca95248774
```
  - ... and 16 more failures.
- Test aes_ctr_fi has 1 failures.
  - 28.aes_ctr_fi.72675715204250725277653192912276635672025797954731377178500185802068276373297
    Log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/28.aes_ctr_fi/latest/run.log
```
  Job ID: smart:0e699aec-8252-4155-b9c0-02c975c06cea
```

UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 7 failures:

Test aes_stress_all_with_rand_reset has 7 failures.

1.aes_stress_all_with_rand_reset.107316727566702841561592457871303076782642028851457205029736825047487639365384
Line 904, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/1.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 238297891 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 238297891 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

2.aes_stress_all_with_rand_reset.86556345141442882252315731538047700706045983540870623896789907516232159248273
Line 794, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/2.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 454444460 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_reseed_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 454444460 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 5 more failures.

UVM_FATAL (aes_cipher_fi_vseq.sv:62) [aes_cipher_fi_vseq] wait timeout occurred! has 7 failures:
- Test aes_cipher_fi has 7 failures.
  - 49.aes_cipher_fi.97871179867709404063831828278184831185801251062792076602883491761820060854356
    Line 322, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/49.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10009779158 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10009779158 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 100.aes_cipher_fi.76253215317165798932523136642205754955631840824236018405592436315720643359521
    Line 319, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/100.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 10014797914 ps: (aes_cipher_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_cipher_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10014797914 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 5 more failures.
UVM_FATAL (aes_control_fi_vseq.sv:62) [aes_control_fi_vseq] wait timeout occurred! has 5 failures:
- Test aes_control_fi has 5 failures.
  - 10.aes_control_fi.63775379167863828264672046068772067684461757709980484003595282898493659265470
    Line 324, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/10.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10012542213 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10012542213 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 87.aes_control_fi.9191581948078627664134826008250333156947614684918390031455535008279118821290
    Line 322, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/87.aes_control_fi/latest/run.log
```
  UVM_FATAL @ 10003324829 ps: (aes_control_fi_vseq.sv:62) [uvm_test_top.env.virtual_sequencer.aes_control_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10003324829 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 3 more failures.

Exit reason: Error: User command failed UVM_ERROR (uvm_sequencer_base.svh:757) sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'sequencer' for sequence 'sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues has 3 failures:

Test aes_stress_all_with_rand_reset has 3 failures.

0.aes_stress_all_with_rand_reset.18032179919464822659572685604045297904524723145723998435006097158455014777430
Line 1166, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/0.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 3138026693 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_stress_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 3138026693 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

7.aes_stress_all_with_rand_reset.11206711445233144593310770745516523614518442788420574892932891305684231146237
Line 607, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/7.aes_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 3091095610 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.keymgr_sideload_agent.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.keymgr_sideload_agent.sequencer' for sequence 'uvm_test_top.env.virtual_sequencer.aes_stress_vseq.sideload_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues
  UVM_INFO @ 3091095610 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 1 more failures.

Exit reason: Error: User command failed xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_*/rtl/aes_core.sv,987): Assertion AesSecCmDataRegLocalEscDataOut has failed (* cycles, starting * PS) has 2 failures:

Test aes_stress_all has 1 failures.

7.aes_stress_all.102660292789001822321924177048219955056977059533650373551836541295167055274382
Line 72396, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/7.aes_stress_all/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 1005511675 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 1005471675 PS)
  UVM_ERROR @ 1005511675 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut
  UVM_INFO @ 1005511675 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test aes_alert_reset has 1 failures.

20.aes_alert_reset.30349814025771308840224712802759653707756899386046603316993172922320806330161
Line 1355, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/20.aes_alert_reset/latest/run.log

  xmsim: *E,ASRTST (/workspace/default/src/lowrisc_ip_aes_1.0/rtl/aes_core.sv,987): (time 10083051 PS) Assertion tb.dut.u_aes_core.AesSecCmDataRegLocalEscDataOut has failed (2 cycles, starting 10056024 PS)
  UVM_ERROR @  10083051 ps: (aes_core.sv:987) [ASSERT FAILED] AesSecCmDataRegLocalEscDataOut
  UVM_INFO @  10083051 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_FATAL (aes_core_fi_vseq.sv:66) [aes_core_fi_vseq] wait timeout occurred! has 2 failures:
- Test aes_core_fi has 2 failures.
  - 43.aes_core_fi.102915639187461561848259874271890203366344036902868953668669159671589688567048
    Line 319, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/43.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10016568690 ps: (aes_core_fi_vseq.sv:66) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10016568690 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 45.aes_core_fi.111195108736395747300301960863717440012590180850352809972031139437836420454056
    Line 314, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/45.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10008672352 ps: (aes_core_fi_vseq.sv:66) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10008672352 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_fi_vseq.sv:69) virtual_sequencer [aes_fi_vseq] Was Able to finish without clearing reset has 1 failures:
- Test aes_fi has 1 failures.
  - 16.aes_fi.23116816730958823604075403205552968449545289214215892190843018558502596539936
    Line 2458, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/16.aes_fi/latest/run.log
```
  UVM_FATAL @  49469622 ps: (aes_fi_vseq.sv:69) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.aes_fi_vseq] Was Able to finish without clearing reset
  UVM_INFO @  49469622 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=*, Comparison=CompareOpEq, exp_data=*, call_count=11) has 1 failures:
- Test aes_core_fi has 1 failures.
  - 16.aes_core_fi.705274400554712160822231117542746576594818419129186888853028069121908232782
    Line 320, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/16.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10042845101 ps: (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=0x8b3e8084, Comparison=CompareOpEq, exp_data=0x0, call_count=11)
  UVM_INFO @ 10042845101 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (aes_core_fi_vseq.sv:89) [aes_core_fi_vseq] wait timeout occurred! has 1 failures:
- Test aes_core_fi has 1 failures.
  - 37.aes_core_fi.72712655817664811392846021227291858344752854548777832187581444510320635842069
    Line 315, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/37.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10027791552 ps: (aes_core_fi_vseq.sv:89) [uvm_test_top.env.virtual_sequencer.aes_core_fi_vseq] wait timeout occurred!
  UVM_INFO @ 10027791552 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=*, Comparison=CompareOpEq, exp_data=*, call_count=9) has 1 failures:
- Test aes_core_fi has 1 failures.
  - 54.aes_core_fi.105706715734010409976189981997174330005102534660976785332670410697720050311205
    Line 314, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/54.aes_core_fi/latest/run.log
```
  UVM_FATAL @ 10007934952 ps: (csr_utils_pkg.sv:587) [csr_utils::csr_spinwait] timeout aes_reg_block.status.idle (addr=0x8df73684, Comparison=CompareOpEq, exp_data=0x0, call_count=9)
  UVM_INFO @ 10007934952 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (uvm_phase.svh:1521) [PH_TIMEOUT] Explicit timeout of * ps hit, indicating a probable testbench issue has 1 failures:
- Test aes_cipher_fi has 1 failures.
  - 297.aes_cipher_fi.2736327039129360538548453399156812586796052592811438666278389807301385397301
    Line 317, in log /container/opentitan-public/scratch/os_regression/aes_unmasked-sim-xcelium/297.aes_cipher_fi/latest/run.log
```
  UVM_FATAL @ 200000000000 ps: (uvm_phase.svh:1521) [PH_TIMEOUT] Explicit timeout of 200000000000 ps hit, indicating a probable testbench issue
  UVM_INFO @ 200000000000 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```