SRAM_CTRL/RET Simulation Results

Tuesday May 28 2024 19:30:06 UTC

GitHub Revision: `0e5093d709`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 51604449886868634540233838791789448907774502353938218657919214072353062987195

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	sram_ctrl_smoke	3.018m	1.370ms	49	50	98.00
V1	csr_hw_reset	sram_ctrl_csr_hw_reset	0.700s	25.013us	5	5	100.00
V1	csr_rw	sram_ctrl_csr_rw	0.770s	12.965us	20	20	100.00
V1	csr_bit_bash	sram_ctrl_csr_bit_bash	2.120s	552.975us	5	5	100.00
V1	csr_aliasing	sram_ctrl_csr_aliasing	0.770s	22.700us	5	5	100.00
V1	csr_mem_rw_with_rand_reset	sram_ctrl_csr_mem_rw_with_rand_reset	2.770s	154.149us	18	20	90.00
V1	regwen_csr_and_corresponding_lockable_csr	sram_ctrl_csr_rw	0.770s	12.965us	20	20	100.00
		sram_ctrl_csr_aliasing	0.770s	22.700us	5	5	100.00
V1	mem_walk	sram_ctrl_mem_walk	11.630s	663.112us	50	50	100.00
V1	mem_partial_access	sram_ctrl_mem_partial_access	6.870s	170.902us	50	50	100.00
V1		TOTAL			202	205	98.54
V2	multiple_keys	sram_ctrl_multiple_keys	35.014m	195.995ms	50	50	100.00
V2	stress_pipeline	sram_ctrl_stress_pipeline	7.772m	35.013ms	50	50	100.00
V2	bijection	sram_ctrl_bijection	1.608m	22.665ms	50	50	100.00
V2	access_during_key_req	sram_ctrl_access_during_key_req	2.870m	6.008ms	1	50	2.00
V2	lc_escalation	sram_ctrl_lc_escalation	10.150s	2.553ms	50	50	100.00
V2	executable	sram_ctrl_executable	30.339m	18.527ms	50	50	100.00
V2	partial_access	sram_ctrl_partial_access	2.303m	3.477ms	49	50	98.00
		sram_ctrl_partial_access_b2b	9.351m	49.013ms	50	50	100.00
V2	max_throughput	sram_ctrl_max_throughput	2.614m	1.826ms	50	50	100.00
		sram_ctrl_throughput_w_partial_write	2.758m	426.851us	50	50	100.00
V2	regwen	sram_ctrl_regwen	35.174m	72.178ms	49	50	98.00
V2	ram_cfg	sram_ctrl_ram_cfg	0.840s	88.998us	50	50	100.00
V2	stress_all	sram_ctrl_stress_all	1.674h	44.359ms	2	50	4.00
V2	alert_test	sram_ctrl_alert_test	0.710s	45.538us	50	50	100.00
V2	tl_d_oob_addr_access	sram_ctrl_tl_errors	5.030s	350.052us	20	20	100.00
V2	tl_d_illegal_access	sram_ctrl_tl_errors	5.030s	350.052us	20	20	100.00
V2	tl_d_outstanding_access	sram_ctrl_csr_hw_reset	0.700s	25.013us	5	5	100.00
		sram_ctrl_csr_rw	0.770s	12.965us	20	20	100.00
		sram_ctrl_csr_aliasing	0.770s	22.700us	5	5	100.00
		sram_ctrl_same_csr_outstanding	0.850s	48.791us	20	20	100.00
V2	tl_d_partial_access	sram_ctrl_csr_hw_reset	0.700s	25.013us	5	5	100.00
		sram_ctrl_csr_rw	0.770s	12.965us	20	20	100.00
		sram_ctrl_csr_aliasing	0.770s	22.700us	5	5	100.00
		sram_ctrl_same_csr_outstanding	0.850s	48.791us	20	20	100.00
V2		TOTAL			641	740	86.62
V2S	passthru_mem_tl_intg_err	sram_ctrl_passthru_mem_tl_intg_err	3.850s	2.251ms	19	20	95.00
V2S	tl_intg_err	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
		sram_ctrl_tl_intg_err	2.850s	1.194ms	14	20	70.00
V2S	prim_count_check	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
V2S	sec_cm_bus_integrity	sram_ctrl_tl_intg_err	2.850s	1.194ms	14	20	70.00
V2S	sec_cm_ctrl_config_regwen	sram_ctrl_regwen	35.174m	72.178ms	49	50	98.00
V2S	sec_cm_exec_config_regwen	sram_ctrl_csr_rw	0.770s	12.965us	20	20	100.00
V2S	sec_cm_exec_config_mubi	sram_ctrl_executable	30.339m	18.527ms	50	50	100.00
V2S	sec_cm_exec_intersig_mubi	sram_ctrl_executable	30.339m	18.527ms	50	50	100.00
V2S	sec_cm_lc_hw_debug_en_intersig_mubi	sram_ctrl_executable	30.339m	18.527ms	50	50	100.00
V2S	sec_cm_lc_escalate_en_intersig_mubi	sram_ctrl_lc_escalation	10.150s	2.553ms	50	50	100.00
V2S	sec_cm_mem_integrity	sram_ctrl_passthru_mem_tl_intg_err	3.850s	2.251ms	19	20	95.00
V2S	sec_cm_mem_readback	sram_ctrl_smoke	3.018m	1.370ms	49	50	98.00
V2S	sec_cm_mem_scramble	sram_ctrl_smoke	3.018m	1.370ms	49	50	98.00
V2S	sec_cm_addr_scramble	sram_ctrl_smoke	3.018m	1.370ms	49	50	98.00
V2S	sec_cm_instr_bus_lc_gated	sram_ctrl_executable	30.339m	18.527ms	50	50	100.00
V2S	sec_cm_ram_tl_lc_gate_fsm_sparse	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
V2S	sec_cm_key_global_esc	sram_ctrl_lc_escalation	10.150s	2.553ms	50	50	100.00
V2S	sec_cm_key_local_esc	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
V2S	sec_cm_init_ctr_redun	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
V2S	sec_cm_scramble_key_sideload	sram_ctrl_smoke	3.018m	1.370ms	49	50	98.00
V2S	sec_cm_tlul_fifo_ctr_redun	sram_ctrl_sec_cm	3.480s	3.087ms	5	5	100.00
V2S		TOTAL			38	45	84.44
V3	stress_all_with_rand_reset	sram_ctrl_stress_all_with_rand_reset	10.916m	6.623ms	7	50	14.00
V3		TOTAL			7	50	14.00
		TOTAL			888	1040	85.38

Testplan Progress

Items	Total	Written	Passing	Progress
V1	8	8	6	75.00
V2	16	16	12	75.00
V2S	3	3	1	33.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
98.22	99.18	95.41	100.00	100.00	96.12	99.56	97.26

Failure Buckets

UVM_WARNING (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem. has 89 failures:

Test sram_ctrl_stress_all_with_rand_reset has 40 failures.

0.sram_ctrl_stress_all_with_rand_reset.96164414036681736316070359407975847674270236107104192955219892737000893555708
Line 310, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/0.sram_ctrl_stress_all_with_rand_reset/latest/run.log

  UVM_WARNING @ 738809633 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 739704377 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_executable_vseq] Update READBACK Value
  UVM_INFO @ 740072801 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 906653081 ps: (cip_base_vseq__tl_errors.svh:196) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Running run_tl_errors_vseq 2/136
  UVM_INFO @ 1226918801 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value

1.sram_ctrl_stress_all_with_rand_reset.112755668080895633731718841673433130107627888469439404019317009196921557186327
Line 318, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/1.sram_ctrl_stress_all_with_rand_reset/latest/run.log

  UVM_WARNING @ 340684634 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 340787576 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_stress_all_vseq] Update READBACK Value
  UVM_INFO @ 340846400 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_regwen_vseq] Update READBACK Value
  UVM_INFO @ 360640676 ps: (dv_base_reg.sv:325) [sram_ctrl_regs_reg_block.exec_regwen] lock_lockable_flds          0 val
  UVM_INFO @ 360728912 ps: (dv_base_reg.sv:325) [sram_ctrl_regs_reg_block.ctrl_regwen] lock_lockable_flds          0 val

... and 38 more failures.

Test sram_ctrl_stress_all has 46 failures.

0.sram_ctrl_stress_all.102496370008266280329297862513481744239800854549762087293209478214833900820956
Line 306, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/0.sram_ctrl_stress_all/latest/run.log

  UVM_WARNING @ 649463576 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 649503576 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_stress_all_vseq] Update READBACK Value
  UVM_INFO @ 649523576 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_multiple_keys_vseq] Update READBACK Value
  UVM_INFO @ 649543576 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_lc_escalation_vseq] Update READBACK Value
  UVM_INFO @ 659023576 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_multiple_keys_vseq] Update READBACK Value

1.sram_ctrl_stress_all.14945099721456319389420291624194864243607891724669495668905895727406941720291
Line 466, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/1.sram_ctrl_stress_all/latest/run.log

  UVM_WARNING @ 15278871308 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 15279025156 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_stress_all_vseq] Update READBACK Value
  UVM_INFO @ 15279102080 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_multiple_keys_vseq] Update READBACK Value
  UVM_INFO @ 15320026648 ps: (sram_ctrl_smoke_vseq.sv:101) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_multiple_keys_vseq] Performing 10000 random memory accesses!
  UVM_INFO @ 15498758562 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_stress_all_vseq] Update READBACK Value

... and 44 more failures.

Test sram_ctrl_tl_intg_err has 3 failures.

5.sram_ctrl_tl_intg_err.28692393060202944179502335762131522061243763622511493774994867673939969571474
Line 661, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/5.sram_ctrl_tl_intg_err/latest/run.log

  UVM_WARNING @ 609252179 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 609502181 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 609835517 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 616717193 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 616717193 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value

13.sram_ctrl_tl_intg_err.103645343609129334229109971860916790765447928995933509610700468416083291391041
Line 506, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/13.sram_ctrl_tl_intg_err/latest/run.log

  UVM_WARNING @ 748686918 ps: (uvm_reg_field.svh:1264) [UVM/FLD/SET/BSY] Setting the value of field "en" while containing register "sram_ctrl_regs_reg_block.readback" is being accessed may result in loss of desired field value. A race condition between threads concurrently accessing the register model is the likely cause of the problem.
  UVM_INFO @ 749499418 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 750061918 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 750811918 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value
  UVM_INFO @ 765624418 ps: (sram_ctrl_base_vseq.sv:65) uvm_test_top.env.virtual_sequencer [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Update READBACK Value

... and 1 more failures.

UVM_ERROR (sram_ctrl_scoreboard.sv:422) [scoreboard] Check failed cfg.in_key_req == * (* [*] vs * [*]) No item is accepted during key req has 46 failures:
- Test sram_ctrl_access_during_key_req has 46 failures.
  - 0.sram_ctrl_access_during_key_req.1868220277216092288941276634330771652271045279185344556771734467772594442799
    Line 291, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/0.sram_ctrl_access_during_key_req/latest/run.log
```
  UVM_ERROR @ 2012275702 ps: (sram_ctrl_scoreboard.sv:422) [uvm_test_top.env.scoreboard] Check failed cfg.in_key_req == 0 (1 [0x1] vs 0 [0x0]) No item is accepted during key req
  UVM_INFO @ 2012275702 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 1.sram_ctrl_access_during_key_req.78167062645056432956063126615290914273869483196392247581822748855000096358506
    Line 276, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/1.sram_ctrl_access_during_key_req/latest/run.log
```
  UVM_ERROR @ 768728526 ps: (sram_ctrl_scoreboard.sv:422) [uvm_test_top.env.scoreboard] Check failed cfg.in_key_req == 0 (1 [0x1] vs 0 [0x0]) No item is accepted during key req
  UVM_INFO @ 768728526 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 44 more failures.

UVM_ERROR sequencer [sequencer] get_next_item/try_next_item called twice without item_done or get in between has 3 failures:

Test sram_ctrl_passthru_mem_tl_intg_err has 1 failures.

5.sram_ctrl_passthru_mem_tl_intg_err.65256689123630708314332984155626128303300118148306144767754834630777671461408
Line 388, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/5.sram_ctrl_passthru_mem_tl_intg_err/latest/run.log

  UVM_ERROR @ 4719698672 ps: uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer [uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer] get_next_item/try_next_item called twice without item_done or get in between
  UVM_INFO @ 4719698672 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test sram_ctrl_tl_intg_err has 2 failures.

9.sram_ctrl_tl_intg_err.77889048726339548961384918571896410258121698191124751005615243643021482676642
Line 589, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/9.sram_ctrl_tl_intg_err/latest/run.log

  UVM_ERROR @  99950694 ps: uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer [uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer] get_next_item/try_next_item called twice without item_done or get in between
  UVM_INFO @  99950694 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

18.sram_ctrl_tl_intg_err.49148085502581807736654844963012851184306294249951437347032064063850334092430
Line 505, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/18.sram_ctrl_tl_intg_err/latest/run.log

  UVM_ERROR @ 397417293 ps: uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer [uvm_test_top.env.m_tl_agent_sram_ctrl_regs_reg_block.sequencer] get_next_item/try_next_item called twice without item_done or get in between
  UVM_INFO @ 397417293 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (cip_base_vseq.sv:829) [sram_ctrl_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 3 failures:
- Test sram_ctrl_stress_all_with_rand_reset has 3 failures.
  - 8.sram_ctrl_stress_all_with_rand_reset.1090678714688588098321799071670872475820430875052241398016922514479414131131
    Line 299, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/8.sram_ctrl_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 648429839 ps: (cip_base_vseq.sv:829) [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Check failed (!has_outstanding_access()) Waited 50000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 648429839 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 41.sram_ctrl_stress_all_with_rand_reset.30400500015685394313067188127083965409833666425838604925874433777946983911393
    Line 294, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/41.sram_ctrl_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 623971529 ps: (cip_base_vseq.sv:829) [uvm_test_top.env.virtual_sequencer.sram_ctrl_common_vseq] Check failed (!has_outstanding_access()) Waited 50000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 623971529 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.
UVM_ERROR (sram_ctrl_scoreboard.sv:441) [scoreboard] Check failed cfg.in_key_req == * (* [*] vs * [*]) No item is accepted during key req has 3 failures:
- Test sram_ctrl_access_during_key_req has 3 failures.
  - 11.sram_ctrl_access_during_key_req.84131526748079021386096526134762713972942108241220103725852612523780594419471
    Line 276, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/11.sram_ctrl_access_during_key_req/latest/run.log
```
  UVM_ERROR @ 144881032 ps: (sram_ctrl_scoreboard.sv:441) [uvm_test_top.env.scoreboard] Check failed cfg.in_key_req == 0 (1 [0x1] vs 0 [0x0]) No item is accepted during key req
  UVM_INFO @ 144881032 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 22.sram_ctrl_access_during_key_req.95140189984866412559087849461914194831331680854015215470916817053933603386335
    Line 277, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/22.sram_ctrl_access_during_key_req/latest/run.log
```
  UVM_ERROR @ 160454073 ps: (sram_ctrl_scoreboard.sv:441) [uvm_test_top.env.scoreboard] Check failed cfg.in_key_req == 0 (1 [0x1] vs 0 [0x0]) No item is accepted during key req
  UVM_INFO @ 160454073 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 1 more failures.
UVM_FATAL (cip_base_vseq.sv:250) [sram_ctrl_smoke_vseq] Timeout waiting tl_access : addr=* has 2 failures:
- Test sram_ctrl_partial_access has 1 failures.
  - 3.sram_ctrl_partial_access.35654976813680554591823516015942569961921249895047265621841610840294276725777
    Line 274, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/3.sram_ctrl_partial_access/latest/run.log
```
  UVM_FATAL @ 10517257085 ps: (cip_base_vseq.sv:250) [uvm_test_top.env.virtual_sequencer.sram_ctrl_smoke_vseq] Timeout waiting tl_access : addr=0x97a82418
  UVM_INFO @ 10517257085 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test sram_ctrl_smoke has 1 failures.
  - 20.sram_ctrl_smoke.1413177521183700997347685183470638416393350879577492372974067667194540000043
    Line 274, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/20.sram_ctrl_smoke/latest/run.log
```
  UVM_FATAL @ 10276617176 ps: (cip_base_vseq.sv:250) [uvm_test_top.env.virtual_sequencer.sram_ctrl_smoke_vseq] Timeout waiting tl_access : addr=0xd13e17fa
  UVM_INFO @ 10276617176 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (cip_base_vseq.sv:250) [sram_ctrl_regwen_vseq] Timeout waiting tl_access : addr=* has 2 failures:
- Test sram_ctrl_stress_all has 1 failures.
  - 18.sram_ctrl_stress_all.75374209400509009268738979925330678960978313551742649135078398130166763217653
    Line 281, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/18.sram_ctrl_stress_all/latest/run.log
```
  UVM_FATAL @ 18621275836 ps: (cip_base_vseq.sv:250) [uvm_test_top.env.virtual_sequencer.sram_ctrl_regwen_vseq] Timeout waiting tl_access : addr=0xf58e6159
  UVM_INFO @ 18621275836 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
- Test sram_ctrl_regwen has 1 failures.
  - 44.sram_ctrl_regwen.29808959462152804980029682277067309611222853042684183446808577190342701866989
    Line 341, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/44.sram_ctrl_regwen/latest/run.log
```
  UVM_FATAL @ 61101346256 ps: (cip_base_vseq.sv:250) [uvm_test_top.env.virtual_sequencer.sram_ctrl_regwen_vseq] Timeout waiting tl_access : addr=0x28af52b1
  UVM_INFO @ 61101346256 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
Offending '((pend_req[*].pend == *'b0) || $test$plusargs("disable_assert_final_checks"))' has 1 failures:
- Test sram_ctrl_tl_intg_err has 1 failures.
  - 1.sram_ctrl_tl_intg_err.13861531552353484418599327943641829196160944106742545020305766104627028102007
    Line 676, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/1.sram_ctrl_tl_intg_err/latest/run.log
```
      Offending '((pend_req[231].pend == 1'b0) || $test$plusargs("disable_assert_final_checks"))'
  UVM_ERROR @ 501233597 ps: (tlul_assert.sv:301) [ASSERT FAILED] noOutstandingReqsAtEndOfSim_A
  UVM_INFO @ 501233597 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (csr_utils_pkg.sv:466) [csr_utils::csr_rd_check] Check failed obs == exp (* [*] vs * [*]) Regname: sram_ctrl_regs_reg_block.status reset value: * has 1 failures:
- Test sram_ctrl_csr_mem_rw_with_rand_reset has 1 failures.
  - 16.sram_ctrl_csr_mem_rw_with_rand_reset.8028541245395924763308311506620929061414617234360924522715760341942445006917
    Line 275, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/16.sram_ctrl_csr_mem_rw_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 137738552 ps: (csr_utils_pkg.sv:466) [csr_utils::csr_rd_check] Check failed obs == exp (40 [0x28] vs 0 [0x0]) Regname: sram_ctrl_regs_reg_block.status reset value: 0x0
  UVM_INFO @ 137738552 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (csr_utils_pkg.sv:466) [csr_utils::csr_rd_check] Check failed obs == exp (* [*] vs * [*]) Regname: sram_ctrl_regs_reg_block.scr_key_rotated.success reset value: * has 1 failures:
- Test sram_ctrl_csr_mem_rw_with_rand_reset has 1 failures.
  - 19.sram_ctrl_csr_mem_rw_with_rand_reset.59672021443828970424010139252924286432137860274781142017047180106699037549767
    Line 274, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/19.sram_ctrl_csr_mem_rw_with_rand_reset/latest/run.log
```
  UVM_ERROR @  30136940 ps: (csr_utils_pkg.sv:466) [csr_utils::csr_rd_check] Check failed obs == exp (8 [0x8] vs 9 [0x9]) Regname: sram_ctrl_regs_reg_block.scr_key_rotated.success reset value: 0x9
  UVM_INFO @  30136940 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_FATAL (cip_base_vseq.sv:250) [sram_ctrl_multiple_keys_vseq] Timeout waiting tl_access : addr=* has 1 failures:
- Test sram_ctrl_stress_all has 1 failures.
  - 20.sram_ctrl_stress_all.36545233782076438464980640500337276545532657283742087978404629578593482051612
    Line 309, in log /container/opentitan-public/scratch/os_regression/sram_ctrl_ret-sim-vcs/20.sram_ctrl_stress_all/latest/run.log
```
  UVM_FATAL @ 22475367583 ps: (cip_base_vseq.sv:250) [uvm_test_top.env.virtual_sequencer.sram_ctrl_multiple_keys_vseq] Timeout waiting tl_access : addr=0x7fc461d2
  UVM_INFO @ 22475367583 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```