KEYMGR Simulation Results

Monday May 29 2023 07:02:33 UTC

GitHub Revision: `877a77116`

Branch: os_regression

Testplan

Simulator: VCS

Build randomization enabled with --build-seed 88555427

Test Results

Stage	Name	Tests	Max Job Runtime	Simulated Time	Passing	Total	Pass Rate
V1	smoke	keymgr_smoke	39.900s	1.708ms	50	50	100.00
V1	random	keymgr_random	1.852m	9.521ms	50	50	100.00
V1	csr_hw_reset	keymgr_csr_hw_reset	1.450s	243.092us	5	5	100.00
V1	csr_rw	keymgr_csr_rw	1.550s	29.011us	20	20	100.00
V1	csr_bit_bash	keymgr_csr_bit_bash	14.810s	426.681us	5	5	100.00
V1	csr_aliasing	keymgr_csr_aliasing	13.650s	1.288ms	5	5	100.00
V1	csr_mem_rw_with_rand_reset	keymgr_csr_mem_rw_with_rand_reset	1.630s	42.905us	20	20	100.00
V1	regwen_csr_and_corresponding_lockable_csr	keymgr_csr_rw	1.550s	29.011us	20	20	100.00
		keymgr_csr_aliasing	13.650s	1.288ms	5	5	100.00
V1		TOTAL			155	155	100.00
V2	cfgen_during_op	keymgr_cfg_regwen	1.877m	9.594ms	50	50	100.00
V2	sideload	keymgr_sideload	53.330s	7.708ms	50	50	100.00
		keymgr_sideload_kmac	42.330s	21.070ms	50	50	100.00
		keymgr_sideload_aes	50.590s	1.673ms	50	50	100.00
		keymgr_sideload_otbn	1.042m	1.912ms	50	50	100.00
V2	direct_to_disabled_state	keymgr_direct_to_disabled	19.390s	3.496ms	50	50	100.00
V2	lc_disable	keymgr_lc_disable	9.410s	4.544ms	49	50	98.00
V2	kmac_error_response	keymgr_kmac_rsp_err	1.232m	3.752ms	40	50	80.00
V2	invalid_sw_input	keymgr_sw_invalid_input	1.118m	10.587ms	50	50	100.00
V2	invalid_hw_input	keymgr_hwsw_invalid_input	1.357m	7.537ms	50	50	100.00
V2	sync_async_fault_cross	keymgr_sync_async_fault_cross	13.530s	625.100us	50	50	100.00
V2	stress_all	keymgr_stress_all	9.148m	32.022ms	48	50	96.00
V2	intr_test	keymgr_intr_test	0.880s	56.153us	50	50	100.00
V2	alert_test	keymgr_alert_test	1.030s	38.143us	50	50	100.00
V2	tl_d_oob_addr_access	keymgr_tl_errors	4.290s	1.156ms	20	20	100.00
V2	tl_d_illegal_access	keymgr_tl_errors	4.290s	1.156ms	20	20	100.00
V2	tl_d_outstanding_access	keymgr_csr_hw_reset	1.450s	243.092us	5	5	100.00
		keymgr_csr_rw	1.550s	29.011us	20	20	100.00
		keymgr_csr_aliasing	13.650s	1.288ms	5	5	100.00
		keymgr_same_csr_outstanding	3.560s	191.401us	20	20	100.00
V2	tl_d_partial_access	keymgr_csr_hw_reset	1.450s	243.092us	5	5	100.00
		keymgr_csr_rw	1.550s	29.011us	20	20	100.00
		keymgr_csr_aliasing	13.650s	1.288ms	5	5	100.00
		keymgr_same_csr_outstanding	3.560s	191.401us	20	20	100.00
V2		TOTAL			727	740	98.24
V2S	sec_cm_additional_check	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	tl_intg_err	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
		keymgr_tl_intg_err	32.450s	2.680ms	20	20	100.00
V2S	shadow_reg_update_error	keymgr_shadow_reg_errors	33.490s	9.399ms	20	20	100.00
V2S	shadow_reg_read_clear_staged_value	keymgr_shadow_reg_errors	33.490s	9.399ms	20	20	100.00
V2S	shadow_reg_storage_error	keymgr_shadow_reg_errors	33.490s	9.399ms	20	20	100.00
V2S	shadowed_reset_glitch	keymgr_shadow_reg_errors	33.490s	9.399ms	20	20	100.00
V2S	shadow_reg_update_error_with_csr_rw	keymgr_shadow_reg_errors_with_csr_rw	16.820s	2.873ms	20	20	100.00
V2S	prim_count_check	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	prim_fsm_check	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_bus_integrity	keymgr_tl_intg_err	32.450s	2.680ms	20	20	100.00
V2S	sec_cm_config_shadow	keymgr_shadow_reg_errors	33.490s	9.399ms	20	20	100.00
V2S	sec_cm_op_config_regwen	keymgr_cfg_regwen	1.877m	9.594ms	50	50	100.00
V2S	sec_cm_reseed_config_regwen	keymgr_random	1.852m	9.521ms	50	50	100.00
		keymgr_csr_rw	1.550s	29.011us	20	20	100.00
V2S	sec_cm_sw_binding_config_regwen	keymgr_random	1.852m	9.521ms	50	50	100.00
		keymgr_csr_rw	1.550s	29.011us	20	20	100.00
V2S	sec_cm_max_key_ver_config_regwen	keymgr_random	1.852m	9.521ms	50	50	100.00
		keymgr_csr_rw	1.550s	29.011us	20	20	100.00
V2S	sec_cm_lc_ctrl_intersig_mubi	keymgr_lc_disable	9.410s	4.544ms	49	50	98.00
V2S	sec_cm_constants_consistency	keymgr_hwsw_invalid_input	1.357m	7.537ms	50	50	100.00
V2S	sec_cm_intersig_consistency	keymgr_hwsw_invalid_input	1.357m	7.537ms	50	50	100.00
V2S	sec_cm_hw_key_sw_noaccess	keymgr_random	1.852m	9.521ms	50	50	100.00
V2S	sec_cm_output_keys_ctrl_redun	keymgr_sideload_protect	19.860s	769.124us	50	50	100.00
V2S	sec_cm_ctrl_fsm_sparse	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_data_fsm_sparse	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_local_esc	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_ctrl_fsm_consistency	keymgr_custom_cm	1.294m	2.229ms	49	50	98.00
V2S	sec_cm_ctrl_fsm_global_esc	keymgr_lc_disable	9.410s	4.544ms	49	50	98.00
V2S	sec_cm_ctrl_ctr_redun	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_kmac_if_fsm_sparse	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_kmac_if_ctr_redun	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_kmac_if_cmd_ctrl_consistency	keymgr_custom_cm	1.294m	2.229ms	49	50	98.00
V2S	sec_cm_kmac_if_done_ctrl_consistency	keymgr_custom_cm	1.294m	2.229ms	49	50	98.00
V2S	sec_cm_reseed_ctr_redun	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_side_load_sel_ctrl_consistency	keymgr_custom_cm	1.294m	2.229ms	49	50	98.00
V2S	sec_cm_sideload_ctrl_fsm_sparse	keymgr_sec_cm	3.407m	17.522ms	5	5	100.00
V2S	sec_cm_ctrl_key_integrity	keymgr_custom_cm	1.294m	2.229ms	49	50	98.00
V2S		TOTAL			164	165	99.39
V3	stress_all_with_rand_reset	keymgr_stress_all_with_rand_reset	19.410s	2.466ms	42	50	84.00
V3		TOTAL			42	50	84.00
		TOTAL			1088	1110	98.02

Testplan Progress

Items	Total	Written	Passing	Progress
V1	7	7	7	100.00
V2	16	16	13	81.25
V2S	6	6	5	83.33
V3	1	1	0	0.00

Coverage Results

Coverage Dashboard

SCORE	LINE	COND	TOGGLE	FSM	BRANCH	ASSERT	GROUP
97.77	99.09	98.09	98.14	100.00	99.08	98.38	91.63

Failure Buckets

UVM_ERROR (keymgr_scoreboard.sv:1012) [scoreboard] Check failed act != exp (* [*] vs * [*]) cdi_type: Attestation has 10 failures:

Test keymgr_kmac_rsp_err has 10 failures.

3.keymgr_kmac_rsp_err.3217513361
Line 401, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/3.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @ 178881224 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (1415831882541953152393476046245724609909143940539527975396551656179991295531916633465292502636806019934142116790744113096774469982978142194133794178643542231647529152617144835988704007700141263153142574509188382166119946273388372702337032308150665814329622301560852032455460165548197722553747310674467245632580262476938484984768327343441230475350838751211166290577190820719054612352020025820523042817523493526908575959321243 [0x332b9732ce54365202e3f3112d6a04cc52df74437ddbd63cc3130092955567c43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f97491bb715b31ad3100d77bf936ab5fdbe91144841f1617d5b32a20c42edb3d92a9723fbae947cfc1f2bdb8268f8a577a6bb1fe36cb1a6dbca695bb2b34115ffc3956915208966098ce6d5b8bfb197cf0c5061f8e341073a546567ceff9b4d896f54cc978e5139658077bd04bb09d8a9b] vs 1415831882541953152393476046245724609909143940539527975396551656179991295531916633465292502636806019934142116790744113096774469982978142194133794178643542231647529152617144835988704007700141263153142574509188382166119946273388372702337032308150665814329622301560852032455460165548197722553747310674467245632580262476938484984768327343441230475350838751211166290577190820719054612352020025820523042817523493526908575959321243 [0x332b9732ce54365202e3f3112d6a04cc52df74437ddbd63cc3130092955567c43a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f97491bb715b31ad3100d77bf936ab5fdbe91144841f1617d5b32a20c42edb3d92a9723fbae947cfc1f2bdb8268f8a577a6bb1fe36cb1a6dbca695bb2b34115ffc3956915208966098ce6d5b8bfb197cf0c5061f8e341073a546567ceff9b4d896f54cc978e5139658077bd04bb09d8a9b]) cdi_type: Attestation
  
   DiversificationKey act: 0xc5061f8e341073a546567ceff9b4d896f54cc978e5139658077bd04bb09d8a9b, exp: 0xc5061f8e341073a546567ceff9b4d896f54cc978e5139658077bd04bb09d8a9b
   RomDigest act: 0x6bb1fe36cb1a6dbca695bb2b34115ffc3956915208966098ce6d5b8bfb197cf0, exp: 0x6bb1fe36cb1a6dbca695bb2b34115ffc3956915208966098ce6d5b8bfb197cf0
   HealthMeasurement act: 0xa9723fbae947cfc1f2bdb8268f8a577a, exp: 0xa9723fbae947cfc1f2bdb8268f8a577a

6.keymgr_kmac_rsp_err.3023562395
Line 504, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/6.keymgr_kmac_rsp_err/latest/run.log

  UVM_ERROR @  55562413 ps: (keymgr_scoreboard.sv:1012) [uvm_test_top.env.scoreboard] Check failed act != exp (1519875083729902432358792175442088736900743030251045349375229049185869030446620868670847216298818411241723565063799120262389872611936155493486970034180524064830738669055499651917735611769199171077850154952587091876886034749457484439762228797875430381788887758854510186380352308207640153244736071872804905457484113648783895234359261107922285089134455471879769936973574374289403819526391113281468964305443746114975942189953379 [0x36ee38ad4572c1ac69b8f799a47c3e45ce1971d9501008f32434e7c3bc8bc1783a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9053ae6f16412c45dc0cee54f3844135a08d822a9b24d73f8a157a7228fdefd76b2dcdca04ac6e329b26163b6f0358cae3175506a7ff9aa3a30ee50134800dafb1484145d1c4f3cbac2cea2213c9188e8cdd771249e5d53536035b903c75ad0916a21786d87fdbcc5244d2d2e193ab163] vs 1519875083729902432358792175442088736900743030251045349375229049185869030446620868670847216298818411241723565063799120262389872611936155493486970034180524064830738669055499651917735611769199171077850154952587091876886034749457484439762228797875430381788887758854510186380352308207640153244736071872804905457484113648783895234359261107922285089134455471879769936973574374289403819526391113281468964305443746114975942189953379 [0x36ee38ad4572c1ac69b8f799a47c3e45ce1971d9501008f32434e7c3bc8bc1783a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9053ae6f16412c45dc0cee54f3844135a08d822a9b24d73f8a157a7228fdefd76b2dcdca04ac6e329b26163b6f0358cae3175506a7ff9aa3a30ee50134800dafb1484145d1c4f3cbac2cea2213c9188e8cdd771249e5d53536035b903c75ad0916a21786d87fdbcc5244d2d2e193ab163]) cdi_type: Attestation
  
   DiversificationKey act: 0xcdd771249e5d53536035b903c75ad0916a21786d87fdbcc5244d2d2e193ab163, exp: 0xcdd771249e5d53536035b903c75ad0916a21786d87fdbcc5244d2d2e193ab163
   RomDigest act: 0x3175506a7ff9aa3a30ee50134800dafb1484145d1c4f3cbac2cea2213c9188e8, exp: 0x3175506a7ff9aa3a30ee50134800dafb1484145d1c4f3cbac2cea2213c9188e8
   HealthMeasurement act: 0xb2dcdca04ac6e329b26163b6f0358cae, exp: 0xb2dcdca04ac6e329b26163b6f0358cae

... and 8 more failures.

UVM_ERROR (keymgr_scoreboard.sv:674) [scoreboard] Check failed item.d_data == addr_phase_op_status (* [*] vs * [*]) has 5 failures:

Test keymgr_stress_all_with_rand_reset has 5 failures.

14.keymgr_stress_all_with_rand_reset.188601356
Line 730, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/14.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 1067176625 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 1067176625 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

34.keymgr_stress_all_with_rand_reset.2916171846
Line 1142, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/34.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 570900161 ps: (keymgr_scoreboard.sv:674) [uvm_test_top.env.scoreboard] Check failed item.d_data == addr_phase_op_status (1 [0x1] vs 2 [0x2])
  UVM_INFO @ 570900161 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

... and 3 more failures.

UVM_ERROR (cip_base_scoreboard.sv:281) scoreboard [scoreboard] alert recov_operation_err did not trigger max_delay:* has 4 failures:

Test keymgr_custom_cm has 1 failures.

16.keymgr_custom_cm.4046216147
Line 277, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/16.keymgr_custom_cm/latest/run.log

  UVM_ERROR @  13224315 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  13224315 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_lc_disable has 1 failures.

17.keymgr_lc_disable.3899999803
Line 382, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/17.keymgr_lc_disable/latest/run.log

  UVM_ERROR @  65872724 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:2000
  UVM_INFO @  65872724 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Test keymgr_stress_all has 2 failures.

24.keymgr_stress_all.436173489
Line 321, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_stress_all/latest/run.log

  UVM_ERROR @  80071639 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  80071639 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

43.keymgr_stress_all.1596608473
Line 311, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/43.keymgr_stress_all/latest/run.log

  UVM_ERROR @  30034203 ps: (cip_base_scoreboard.sv:281) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] alert recov_operation_err did not trigger max_delay:0
  UVM_INFO @  30034203 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

UVM_ERROR (keymgr_scoreboard.sv:1010) [scoreboard] Check failed act == exp (* [*] vs * [*]) cdi_type: Attestation has 3 failures:

Test keymgr_stress_all_with_rand_reset has 3 failures.

17.keymgr_stress_all_with_rand_reset.818364369
Line 776, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/17.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @  55343400 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (6106632148589598105433493772509817271260626641795823749383339615972739857857877781411662074862257482306197284364827623154379674730890350453205264096494760547573494715864673588383254172920725588552536262570655166990313266049185940554520371448442179204893681794867207440398204643086131568132886846240246831980817401621499526738342782048661646123115184276778350432270508558432900283793754712527277964011402517319696768687818342 [0xdcb3ea4634877836bddf4e143e7f43e6a36b983f2bc6b6b76b40dbebeabf1dab3a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ae414e0723297db7926b2a712bf724d6f8d3571f4971b53a8ccefceeaac148e1513db99e499c0dad1d8d90ec625b4f204ba3f2046ef53b0e4fb174e553cfc98230e9b1c0ee382300bf4857fc46bbc9b3d5bc17f1c9bd5a4d0ef754539f5c80bbe1037c73be12423deb127d29dcae4666] vs 4331336305624194033286850225433603258610729577846549301238139871028116059473341200054090683707819830545076605950814637936538603055135732677902976939043984033562700530449504237537719386896831523802198438953799848087320881598572776799781832935047489824165963561212274841302297467448651666462642237570067326911920105336889019004124200668098156049843359294412953544441998727830447693700984432325377559596555549319873249512670822 [0x9c8a7bf5ec1692dbc4563dfcfe9575be7ff01ea982862d2c18440c2e308edd293a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f9ae414e0723297db7926b2a712bf724d6f8d3571f4971b53a8ccefceeaac148e1513db99e499c0dad1d8d90ec625b4f204ba3f2046ef53b0e4fb174e553cfc98230e9b1c0ee382300bf4857fc46bbc9b3d5bc17f1c9bd5a4d0ef754539f5c80bbe1037c73be12423deb127d29dcae4666]) cdi_type: Attestation
  
   DiversificationKey act: 0xd5bc17f1c9bd5a4d0ef754539f5c80bbe1037c73be12423deb127d29dcae4666, exp: 0xd5bc17f1c9bd5a4d0ef754539f5c80bbe1037c73be12423deb127d29dcae4666
   RomDigest act: 0x4ba3f2046ef53b0e4fb174e553cfc98230e9b1c0ee382300bf4857fc46bbc9b3, exp: 0x4ba3f2046ef53b0e4fb174e553cfc98230e9b1c0ee382300bf4857fc46bbc9b3
   HealthMeasurement act: 0x513db99e499c0dad1d8d90ec625b4f20, exp: 0x513db99e499c0dad1d8d90ec625b4f20

24.keymgr_stress_all_with_rand_reset.3849634955
Line 1313, in log /container/opentitan-public/scratch/os_regression/keymgr-sim-vcs/24.keymgr_stress_all_with_rand_reset/latest/run.log

  UVM_ERROR @ 632064989 ps: (keymgr_scoreboard.sv:1010) [uvm_test_top.env.scoreboard] Check failed act == exp (2200872281792493163560472784076772806906812453693540346917496786808833136169827841796549181423822787185716447226555936516803123715582421019211459524776787704507729496715349847106755934105014408673566905013893029842998230246295989529427573367464337523896497909273587818263320608887874434948632613584448996598609277785236349088878654414793832610447777195245849277167081126831345413952281538223852334978475787898603661646921466 [0x4f8af56198f1573b50691b391eefde31ac1445a173eff3fade6f846e5f30f9293a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f953b4ab5dae8e869f66ce30d9cc9c1fff72ee0af8a20cc26259d29aa53ec9c6769f6f650ba4aad77e53e0cef414af5a38d93ea1fd8ce4afb61dd04bb1c9addba4c35b78d068c594b52357fcf7d90b2975801047d1b90dfd6ac0aae8f27e59d9fb9b8284449aaeca45eba51e6f5ba07efa] vs 6091709775377973809780717639940223172973058096809771615076942433259991901563443846651442674567800313031057164806605793493363954751856095661263195806693210384253622460432120977561626218965981811463281825252316593319498186276769890818010685602071211905411237888190199113181338656689112934418756366765604205618505356289539233171091614057495141548207456927400909381773225542769690130545458564359813935078199593098503506878299898 [0xdc29d99b5cf2909e7b573ec6a1b5a94867c47ff867023c08fa41a63f3ad190983a0a6d73cd50897de4d744bd65ebdb3837ea77087d878651c517c18a5742b2f953b4ab5dae8e869f66ce30d9cc9c1fff72ee0af8a20cc26259d29aa53ec9c6769f6f650ba4aad77e53e0cef414af5a38d93ea1fd8ce4afb61dd04bb1c9addba4c35b78d068c594b52357fcf7d90b2975801047d1b90dfd6ac0aae8f27e59d9fb9b8284449aaeca45eba51e6f5ba07efa]) cdi_type: Attestation
  
   DiversificationKey act: 0x801047d1b90dfd6ac0aae8f27e59d9fb9b8284449aaeca45eba51e6f5ba07efa, exp: 0x801047d1b90dfd6ac0aae8f27e59d9fb9b8284449aaeca45eba51e6f5ba07efa
   RomDigest act: 0xd93ea1fd8ce4afb61dd04bb1c9addba4c35b78d068c594b52357fcf7d90b2975, exp: 0xd93ea1fd8ce4afb61dd04bb1c9addba4c35b78d068c594b52357fcf7d90b2975
   HealthMeasurement act: 0x9f6f650ba4aad77e53e0cef414af5a38, exp: 0x9f6f650ba4aad77e53e0cef414af5a38

... and 1 more failures.