Lodestar Holesky Rescue Overview

[philknows]

For Lodestar users who are helping with the Holesky testnet rescue efforts, this issue is to track and create an informational record for troubleshooting Lodestar and provide the most up-to-date knowledge and information about using Lodestar for Holesky recovery purposes.

This issue will be updated with the most recent information.

Releases

As of Mar 12 @ 0200 UTC, the latest recommended release for Holesky usage is:

• ethpandaops/lodestar:holesky-rescue-fb59152 until the release of v1.28.0.

As of Mar 7 @ 1900 UTC, we're serving the following unfinalized checkpoint for faster syncing:
- https://pub-3cd22dca6ea14e7a83ce8eb86685b16e.r2.dev/0x8ace010000000000847adec23f86a48d8b9a1f70b850235cebf966d127e1ac2c0e26e1dbcc047592

Using ethpandaops/lodestar:holesky-rescue-edab7f0 or newer, you can persist your own local unfinalized checkpoints now for your own local use. Please see #7509.

NOTE: If you're already running a Lodestar beacon node for the rescue, we do not recommend updating as turning off your beacon node will make it harder to sync back to head as more time progresses.

For the full changelog please see #7501 .

Recommended Configurations

EL Clients

• We recommend utilizing the latest EL images as stated in this HackMD document by execution layer client teams
• We recommend syncing EL clients with pre-Pectra snapshots by EthPandaOps

Lodestar

• Use the latest release which includes fixes we discover while trying to sync Lodestar as stated above
• Use the --bootnodes flag and connect to ENRs which are synced to the tip of the chain.
• Use --network.connectToDiscv5Bootnodes: true to actually connect to the --bootnodes and not just use them for peer discovery
• If you are stuck on Searching peers..., you may want to use --sync.slotImportTolerance=100 which will then show the node as Synced so you can submit proposals/attestations
• If you have a beacon state SSZ you want to use as a checkpoint, you can use --checkpointState <path-to-file or url to raw ssz file> to import the SSZ file for a trusted checkpoint (it does not need to be a finalized state)
• If you run into OOM issues, it's because your nodeJS env options will limit the amount of memory your Lodestar instance can consume. Increasing --max-old-space-size from the default 16384 (16GB) on your nodeJS environment.
• If you see poor attestations, you can try increasing your peer count by using the --targetPeers flag. Targeting 200 peers may improve your attestations

Tips

Ensure you're on the correct chain

These Holesky releases include a blacklist which will ensure you cannot sync on the invalid chain. The bad block at 3711006 should return null when using the command to check the execution payload blockhash:

curl -s http://localhost:9596/eth/v2/beacon/blocks/3711006 | jq ".data.message.body.execution_payload.block_hash"

Submitting your slashable attestations

If you have Holesky validators which voted for the invalid justified epoch, your validator may prevent you from submitting slashable surround votes. To remove the validator database which is preventing you from submitting these slashable attestations, you will need to delete the folder <dataDir>/validator-db and restart your validator client.

[philknows]

Update from Lodestar side, we have fixed a lot of issues and have 10+ nodes synced to head now, still need to activate more validators but seeing blocks and sync committee messages being produced correctly on those that are active and pointed to a synced beacon node.

Summary of improvements

• reduced memory spikes when processing attester slashings
• improved syncing from pre-electra to head, takes "only" around 1h now
• fixed OOM issue due to checkpoint states not being pruned correctly
• still seeing some synced nodes falling out of sync again, needs further investigation

What you need to do if you run Lodestar

• update to new image ethpandaops/lodestar:holesky-rescue-2698a51, or holesky-rescue branch if you build from source
• make sure to remove <dataDir>/chain-db and <dataDir>/peerstore when updating to latest version, while not strictly necessary, it helps to sync faster if your node was stuck previously
• around 10GB memory usage now, although recommendation is to set --max-old-space-size=32768 (32GB limit) to handle sudden spikes, eg. when slashings start to happen. We have set this limit by default, just make sure you are not overriding it in your setup to a lower value

Let us know if you have any issues

[philknows]

Last update: holesky-rescue branch will be rebased in the future from unstable once we get some bug fixes in unstable which were discovered during holesky non-finality.

For Holesky node operators, please continue using the latest `ethpandaops/lodestar:holesky-rescue-fb59152 until the release of v1.28.0.