KMAC/MASKED Simulation Results

Wednesday July 31 2024 23:02:38 UTC

GitHub Revision: `e9b7e615a7`

Branch: os_regression

Simulator: VCS

Build randomization enabled with --build-seed 25204348267605859133056659113100703417171299070132656462514712657132693373848

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	kmac_smoke	1.730m	18.274ms	50	50	100.00
V1	csr_hw_reset	kmac_csr_hw_reset	1.230s	284.248us	5	5	100.00
V1	csr_rw	kmac_csr_rw	1.290s	269.520us	20	20	100.00
V1	csr_bit_bash	kmac_csr_bit_bash	15.890s	1.180ms	5	5	100.00
V1	csr_aliasing	kmac_csr_aliasing	9.490s	1.679ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	kmac_csr_mem_rw_with_rand_reset	2.740s	460.594us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	kmac_csr_rw	1.290s	269.520us	20	20	100.00
		kmac_csr_aliasing	9.490s	1.679ms	5	5	100.00
V1	mem_walk	kmac_mem_walk	0.770s	26.139us	5	5	100.00
V1	mem_partial_access	kmac_mem_partial_access	1.540s	76.798us	5	5	100.00
V1		TOTAL			115	115	100.00
V2	long_msg_and_output	kmac_long_msg_and_output	54.749m	65.853ms	33	50	66.00
V2	burst_write	kmac_burst_write	30.148m	15.574ms	50	50	100.00
V2	test_vectors	kmac_test_vectors_sha3_224	59.779m	362.632ms	44	50	88.00
		kmac_test_vectors_sha3_256	59.537m	1.481s	48	50	96.00
		kmac_test_vectors_sha3_384	46.787m	92.508ms	49	50	98.00
		kmac_test_vectors_sha3_512	31.634m	50.926ms	50	50	100.00
		kmac_test_vectors_shake_128	1.928h	993.634ms	24	50	48.00
		kmac_test_vectors_shake_256	1.609h	71.251ms	17	50	34.00
		kmac_test_vectors_kmac	7.900s	355.388us	50	50	100.00
		kmac_test_vectors_kmac_xof	7.850s	1.023ms	50	50	100.00
V2	sideload	kmac_sideload	11.168m	23.638ms	50	50	100.00
V2	app	kmac_app	7.678m	29.155ms	48	50	96.00
V2	app_with_partial_data	kmac_app_with_partial_data	7.142m	19.403ms	10	10	100.00
V2	entropy_refresh	kmac_entropy_refresh	7.571m	21.110ms	49	50	98.00
V2	error	kmac_error	9.917m	63.146ms	50	50	100.00
V2	key_error	kmac_key_error	13.950s	7.540ms	50	50	100.00
V2	edn_timeout_error	kmac_edn_timeout_error	49.270s	8.495ms	20	20	100.00
V2	entropy_mode_error	kmac_entropy_mode_error	35.550s	1.036ms	20	20	100.00
V2	entropy_ready_error	kmac_entropy_ready_error	1.571m	46.208ms	10	10	100.00
V2	lc_escalation	kmac_lc_escalation	45.250s	6.431ms	50	50	100.00
V2	stress_all	kmac_stress_all	1.229h	504.193ms	50	50	100.00
V2	intr_test	kmac_intr_test	0.880s	20.383us	50	50	100.00
V2	alert_test	kmac_alert_test	0.890s	51.789us	50	50	100.00
V2	tl_d_oob_addr_access	kmac_tl_errors	3.780s	558.876us	20	20	100.00
V2	tl_d_illegal_access	kmac_tl_errors	3.780s	558.876us	20	20	100.00
V2	tl_d_outstanding_access	kmac_csr_hw_reset	1.230s	284.248us	5	5	100.00
		kmac_csr_rw	1.290s	269.520us	20	20	100.00
		kmac_csr_aliasing	9.490s	1.679ms	5	5	100.00
		kmac_same_csr_outstanding	2.910s	932.790us	20	20	100.00
V2	tl_d_partial_access	kmac_csr_hw_reset	1.230s	284.248us	5	5	100.00
		kmac_csr_rw	1.290s	269.520us	20	20	100.00
		kmac_csr_aliasing	9.490s	1.679ms	5	5	100.00
		kmac_same_csr_outstanding	2.910s	932.790us	20	20	100.00
V2		TOTAL			962	1050	91.62
V2S	shadow_reg_update_error	kmac_shadow_reg_errors	1.390s	201.022us	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	kmac_shadow_reg_errors	1.390s	201.022us	20	20	100.00
V2S	shadow_reg_storage_error	kmac_shadow_reg_errors	1.390s	201.022us	20	20	100.00
V2S	shadowed_reset_glitch	kmac_shadow_reg_errors	1.390s	201.022us	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	kmac_shadow_reg_errors_with_csr_rw	3.380s	1.319ms	20	20	100.00
V2S	tl_intg_err	kmac_sec_cm	2.366m	139.669ms	5	5	100.00
		kmac_tl_intg_err	5.810s	2.794ms	20	20	100.00
V2S	sec_cm_bus_integrity	kmac_tl_intg_err	5.810s	2.794ms	20	20	100.00
V2S	sec_cm_lc_escalate_en_intersig_mubi	kmac_lc_escalation	45.250s	6.431ms	50	50	100.00
V2S	sec_cm_sw_key_key_masking	kmac_smoke	1.730m	18.274ms	50	50	100.00
V2S	sec_cm_key_sideload	kmac_sideload	11.168m	23.638ms	50	50	100.00
V2S	sec_cm_cfg_shadowed_config_shadow	kmac_shadow_reg_errors	1.390s	201.022us	20	20	100.00
V2S	sec_cm_fsm_sparse	kmac_sec_cm	2.366m	139.669ms	5	5	100.00
V2S	sec_cm_ctr_redun	kmac_sec_cm	2.366m	139.669ms	5	5	100.00
V2S	sec_cm_packer_ctr_redun	kmac_sec_cm	2.366m	139.669ms	5	5	100.00
V2S	sec_cm_cfg_shadowed_config_regwen	kmac_smoke	1.730m	18.274ms	50	50	100.00
V2S	sec_cm_fsm_global_esc	kmac_lc_escalation	45.250s	6.431ms	50	50	100.00
V2S	sec_cm_fsm_local_esc	kmac_sec_cm	2.366m	139.669ms	5	5	100.00
V2S	sec_cm_absorbed_ctrl_mubi	kmac_mubi	6.095m	31.457ms	10	10	100.00
V2S	sec_cm_sw_cmd_ctrl_sparse	kmac_smoke	1.730m	18.274ms	50	50	100.00
V2S		TOTAL			75	75	100.00
V3	throughput	kmac_throughput			0	0	--
V3	stress_all_with_rand_reset	kmac_stress_all_with_rand_reset	33.669m	90.452ms	5	10	50.00
V3		TOTAL			5	10	50.00
		TOTAL			1157	1250	92.56

Testplan Progress

Items	Total	Written	Passing	Progress
V1	8	8	8	100.00
V2	25	25	17	68.00
V2S	5	5	5	100.00
V3	2	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
94.19	97.91	92.65	99.51	76.76	95.59	99.05	97.88

Failure Buckets

Job kmac_masked-sim-vcs_run_default killed due to: Exit reason: User job exceeded runlimit: User job timed out has 84 failures:
- Test kmac_test_vectors_shake_256 has 33 failures.
  - 0.kmac_test_vectors_shake_256.31460150276261931686233545452057356323949686009367451831715867804147605384377
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/0.kmac_test_vectors_shake_256/latest/run.log
```
  Job ID: smart:75c2c8a0-536a-4ed2-82d8-7a77f95a4eeb
```
  - 2.kmac_test_vectors_shake_256.113255409231136561298437365225489315827986610119927015287686505634090941882138
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/2.kmac_test_vectors_shake_256/latest/run.log
```
  Job ID: smart:4b594f87-5756-4079-b6b3-15ca74413c64
```
  - ... and 31 more failures.
- Test kmac_long_msg_and_output has 17 failures.
  - 1.kmac_long_msg_and_output.70861378369066023722755001315363406224770172778200835427484962814167117359944
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/1.kmac_long_msg_and_output/latest/run.log
```
  Job ID: smart:ee76eb3a-572e-431c-b737-2f4b5eeb805a
```
  - 5.kmac_long_msg_and_output.112481003566899267444188458906793883595388553423416840779681258170145897530031
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/5.kmac_long_msg_and_output/latest/run.log
```
  Job ID: smart:3c223e72-9a6b-4630-a33d-1606dba23e93
```
  - ... and 15 more failures.
- Test kmac_test_vectors_shake_128 has 26 failures.
  - 2.kmac_test_vectors_shake_128.97322009550391761263436919600726974039338287625875140726522719212085482761150
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/2.kmac_test_vectors_shake_128/latest/run.log
```
  Job ID: smart:4b1fbf75-fc75-42ca-877a-6f7f6fce43a5
```
  - 3.kmac_test_vectors_shake_128.55117902885379890035051291319400979417178968887205185033253954145752083495310
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/3.kmac_test_vectors_shake_128/latest/run.log
```
  Job ID: smart:49356acb-170f-4733-bc27-c877427b6317
```
  - ... and 24 more failures.
- Test kmac_test_vectors_sha3_224 has 6 failures.
  - 4.kmac_test_vectors_sha3_224.12299731881866019616019549760425600790298985027895304743401955288869714165947
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/4.kmac_test_vectors_sha3_224/latest/run.log
```
  Job ID: smart:ac4f2840-7ba7-4834-8173-23fe32f0f00d
```
  - 13.kmac_test_vectors_sha3_224.89847137272411225389610406935271811908215944899648726087718926478780752565667
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/13.kmac_test_vectors_sha3_224/latest/run.log
```
  Job ID: smart:f27e29f4-1531-4c6c-8175-556a7e824da4
```
  - ... and 4 more failures.
- Test kmac_test_vectors_sha3_256 has 2 failures.
  - 25.kmac_test_vectors_sha3_256.86300222543522128332466237267010624850587734252702019389989462439318602289156
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/25.kmac_test_vectors_sha3_256/latest/run.log
```
  Job ID: smart:de451c18-d3f3-4138-bc23-caa2f63a5343
```
  - 30.kmac_test_vectors_sha3_256.109577067250124662508738415398636765401924167116505083749338047877651112637258
    Log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/30.kmac_test_vectors_sha3_256/latest/run.log
```
  Job ID: smart:89978e03-3f20-4722-b2d3-de5b9144c73c
```
UVM_ERROR (cip_base_vseq.sv:836) [kmac_common_vseq] Check failed (!has_outstanding_access()) Waited * cycles to issue a reset with no outstanding accesses. has 4 failures:
- Test kmac_stress_all_with_rand_reset has 4 failures.
  - 1.kmac_stress_all_with_rand_reset.56773540781432899502239284608012043461958849753092195941177940266453014463440
    Line 278, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/1.kmac_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 1785219037 ps: (cip_base_vseq.sv:836) [uvm_test_top.env.virtual_sequencer.kmac_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 1785219037 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - 2.kmac_stress_all_with_rand_reset.113803652540022565035768000003626254609681776975121845977273537165413963335921
    Line 401, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/2.kmac_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 4064459959 ps: (cip_base_vseq.sv:836) [uvm_test_top.env.virtual_sequencer.kmac_common_vseq] Check failed (!has_outstanding_access()) Waited 10000 cycles to issue a reset with no outstanding accesses.
  UVM_INFO @ 4064459959 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
  
  
```
  - ... and 2 more failures.

UVM_FATAL (kmac_scoreboard.sv:1638) [scoreboard] Check failed unmasked_digest[i] == dpi_digest[i] (* [*] vs * [*]) Mismatch between unmasked_digest[*] and dpi_digest[*] has 3 failures:

Test kmac_app has 2 failures.

3.kmac_app.79635525991758580662218581409859163402470576510841042255558130707664069603012
Line 309, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/3.kmac_app/latest/run.log

  UVM_FATAL @ 1741508938 ps: (kmac_scoreboard.sv:1638) [uvm_test_top.env.scoreboard] Check failed unmasked_digest[i] == dpi_digest[i] (185 [0xb9] vs 26 [0x1a]) Mismatch between unmasked_digest[0] and dpi_digest[0]
  UVM_INFO @ 1741508938 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

34.kmac_app.109888515291248342170376657227125565100736656071131436472370530433647532060719
Line 601, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/34.kmac_app/latest/run.log

  UVM_FATAL @ 20184759710 ps: (kmac_scoreboard.sv:1638) [uvm_test_top.env.scoreboard] Check failed unmasked_digest[i] == dpi_digest[i] (103 [0x67] vs 112 [0x70]) Mismatch between unmasked_digest[0] and dpi_digest[0]
  UVM_INFO @ 20184759710 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test kmac_entropy_refresh has 1 failures.

36.kmac_entropy_refresh.32757181152782822246181867130352611645718407427071788979158195999339312489832
Line 669, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/36.kmac_entropy_refresh/latest/run.log

  UVM_FATAL @ 5533428074 ps: (kmac_scoreboard.sv:1638) [uvm_test_top.env.scoreboard] Check failed unmasked_digest[i] == dpi_digest[i] (179 [0xb3] vs 156 [0x9c]) Mismatch between unmasked_digest[0] and dpi_digest[0]
  UVM_INFO @ 5533428074 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (kmac_scoreboard.sv:1188) [scoreboard] Check failed csr.get_mirrored_value() == item.d_data (* [*] vs * [*]) reg name: kmac_reg_block.err_code has 1 failures:
- Test kmac_stress_all_with_rand_reset has 1 failures.
  - 9.kmac_stress_all_with_rand_reset.27838952266198484434058907225564200014552056660838778117166814248067325569034
    Line 1844, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/9.kmac_stress_all_with_rand_reset/latest/run.log
```
  UVM_ERROR @ 90451589105 ps: (kmac_scoreboard.sv:1188) [uvm_test_top.env.scoreboard] Check failed csr.get_mirrored_value() == item.d_data (0 [0x0] vs 2147483768 [0x80000078]) reg name: kmac_reg_block.err_code
  UVM_INFO @ 90451589105 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```
UVM_ERROR (csr_utils_pkg.sv:476) [csr_utils::csr_rd_check] Check failed obs == exp (* [*] vs * [*]) Regname: kmac_reg_block.intr_state.kmac_done reset value: * has 1 failures:
- Test kmac_test_vectors_sha3_384 has 1 failures.
  - 34.kmac_test_vectors_sha3_384.114016238876881213810101122713863249725257488009092462188841847591895887938390
    Line 253, in log /container/opentitan-public/scratch/os_regression/kmac_masked-sim-vcs/34.kmac_test_vectors_sha3_384/latest/run.log
```
  UVM_ERROR @  75617532 ps: (csr_utils_pkg.sv:476) [csr_utils::csr_rd_check] Check failed obs == exp (0 [0x0] vs 1 [0x1]) Regname: kmac_reg_block.intr_state.kmac_done reset value: 0x0
  UVM_INFO @  75617532 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---
```