Restart flat db healing pipeline in case of error

[macfarla]

During testing of 24.12.0-RC2, one node failed the sync during the flat db healing step with a MerkleTrieException. There are no other exceptions in the log.

{"@timestamp":"2024-12-09T09:43:38,974","level":"INFO","thread":"EthScheduler-Services-34 (batchHealAndPersistFlatStorageData)","class":"Pipeline","message":"Unexpected exception in pipeline. Aborting.
Throwable summary: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x7c645ffe7048d604cc92a50f9e033b98e05ff48fe07e6d0223abf58fe274d8ce location 0x040005
	at: org.hyperledger.besu.ethereum.trie.StoredNode.lambda$load$0(StoredNode.java:135)","throwable":""}

Leaving #8015 as the investigation of the root cause of this exception.

This issue is to implement the mitigation which is to restart the pipeline if such an error occurs. Currently a restart is required to recover (likely because you get a new pivot block)

[matkt]

do you have all of the logs since the beginning of the node ? I want to be sure we are not missing an exception that will explain this problem

also a restart will not fix the problem, you need to restart the sync from scratch when you have this issue

[jframe]

I have the logs up until it failed, no errors in the logs though. dev-elc-bu-nb-sepolia-burn-snap-24.12.0-RC2_besu.log

Restarting the node fixed the issue. It started the flat db healing again and then continued with the full sync as normal. I couldn't find those logs unfortunately though.