Roy

Summary

This low-priority release contains several optimisations to improve performance on the newly merged mainnet!

Notable changes include:

• Optional payload pruning to reduce execution node timeouts.
• Optimisations to attestation and block processing.
• More efficient attestation subnet subscriptions.
• New flags to control the use of external block builders for MEV.

There are also a few minor breaking changes, described in Breaking Changes.

Note that there is no v3.1.1 release due to a minor bug that was found and fixed before that release had been fully published.

Payload Pruning

One issue we've noticed on mainnet is that requests to the execution node sometimes time out.

ERRO Error fetching block for peer error: ExecutionLayerErrorPayloadReconstruction(0x4092070250431d93d6b0331ec940bf467407302038347382ab57d4e945527e08, EngineError(Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }))

To reduce the number of requests made to the execution node, we have made payload pruning optional in v3.1.2. Payload pruning is the process by which Lighthouse avoids storing execution payloads in its database. When enabled, Lighthouse will fetch payloads from the execution node on-demand. Although this saves disk space it imposes substantial load on the execution node when a syncing peer requests blocks from Lighthouse.

When upgrading to v3.1.2 you should make a choice to enable or disable payload pruning:

• Enable (default): Lighthouse will run a background pruning job on start-up that permanently removes payloads from its database. This pruning may take up to 30 minutes the first time it runs on machines with slower I/O. Lighthouse will continue to be available for validators while this job runs, but may exhibit slightly degraded performance. All future payloads will be deleted from the database and fetched on-demand if required. Although v3.1.0 and earlier also fetched payloads on-demand, the payloads were still stored in the database due to an oversight. Upgrading to v3.1.2 with pruning enabled will delete them permanently.
• Disable: Opt out of payload pruning with the --prune-payloads false flag on the beacon node. Lighthouse will keep all execution payloads in its database and serve them to peers directly. This will consume slightly more disk space but will reduce load on the execution node. This is recommended for users who have seen frequent timeouts and users running on constrained devices (particularly where I/O is scarce). We hope that this may be particularly helpful to users running Besu, which seems to timeout more frequently.

You may freely switch between --prune-payloads false and the default configuration (equivalent to --prune-payloads true). If you are unsure, we recommend starting out with --prune-payloads false, as any payloads deleted after running with payload pruning enabled will remain deleted. Our medium-term goal is to make payload pruning efficient enough to be usable by all nodes, and we are collaborating with the execution client devs on this.

For more information on the implementation of this feature please see #3565, #3587.

Attestation and Block Processing Optimisations

This release includes a ~20% reduction in the time it takes to compute the merkle root of a beacon block (#3581). Since the merge introduced transactions to the beacon chain, computing the merkle root of a block has become more significant. This optimisation will result in faster block imports and therefore better attestation performance (i.e., more accurate head votes).

Additionally, the cache which stores information required to verify attestations has been modified to prevent duplicate work in some scenarios (#3574). Previously, it was possible for multiple attestations being processed on different threads to load the same BeaconState from disk at the same time. This resulted in excess memory and CPU usage. With the new behaviour, the first attestation will load a single BeaconState and other attestations will wait for that operation to complete.

Subscribe On-Demand

Reduces bandwidth by limiting the time we are subscribed to subnets to a smaller time frame when needed. Upgrades to libp2p have allowed us to reduce our subscription time period.

For more information see #3419.

MEV profit threshold

To give validators greater control over their interaction with external block builders, we have added a new flag --builder-profit-threshold which filters out blocks from builders that pay the proposer less than a threshold value.

Please see the documentation for a full description: https://lighthouse-book.sigmaprime.io/builders.html#builder-profit-threshold

Note that this flag is for the beacon node, as the beacon node has lowest-latency access to both the builder payload and the local payload.

We hope that this feature compensates for the deprecation of --strict-fee-recipient (see below).

Breaking Changes

You can upgrade to Lighthouse v3.1.2 without making any changes to your configuration, although you should be aware of the changes in behaviour listed below.

Downgrading to v3.1.0 after upgrading is also supported, so long as the new --prune-payloads and --builder-profit-threshold flags are removed before switching back to v3.1.0.

Deprecation of --strict-fee-recipient

The --strict-fee-recipient flag for the validator client has been deprecated and no longer has any effect.

We took the decision to deprecate the flag for two reasons:

• There was a bug in the implementation that prevented block proposals before TTD was reached (no longer relevant on mainnet).
• Almost all block builders will use a transaction to pay the proposer, meaning that their blocks would be rejected if the strict-fee-recipient flag were used. This means the flag effectively disabled the external block builders entirely, which is not useful.

If you are using --strict-fee-recipient in your Lighthouse VC flags you should remove it. However, if you don't remove it Lighthouse will still start, but will log a warning. In future we will likely remove the flag entirely.

Changes to block reward and block production APIs

Two breaking changes were made to non-standard APIs and parameters:

• The /eth/v2/validator/blocks/{slot} endpoint now uses the standard skip_randao_verification parameter rather than the previously Lighthouse-only parameter verify_randao.
• The POST /lighthouse/analysis/block_rewards now accepts blinded blocks for improved efficiency and better compatibility with MEV builders.

We expect these changes will only affect a small number of expert users. For more information please see #3540.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

The Beacon Node may be updated without the Validator Client, however both components must be updated to support the merge. Beacon nodes and validator clients on pre-v3.0.0 versions have been unable to follow the chain since September 6 2022 when the Bellatrix hard fork was activated.

See Update Priorities for more information about this table.

All Changes

• v3.1.2 (#3603)
• New rust lints for rustc 1.64.0 (#3602)
• send attnet unsubscription event on random subnet expiry (#3600)
• Make garbage collection test less failure prone (#3599)
• Fix ee integration tests (#3592)
• Deduplicate block root computation (#3590)
• Add disable-log-timestamp flag (#3101) (#3586)
• v3.1.1 (#3585)
• Fix concurrency issue with oneshot_broadcast (#3596)
• Impl oneshot_broadcast for committee promises (#3595)
• Avoid holding write-lock whilst waiting on shuffling cache promise (#3589)
• Refined payload pruning (#3587)
• Implement skip_randao_verification and blinded block rewards API (#3540)
• Prune finalized execution payloads (#3565)
• Pre-allocate vectors in SSZ decoding (#3417)
• Use SmallVec for TreeHash packed encoding (#3581)
• Add lcli block-root tool (#3580)
• Avoid duplicate committee cache loads (#3574)
• Add metric for re-org distance (#3566)
• Bump axum deps (#3570)
• Fix builder gas limit docs (#3569)
• Fix ganache test endpoint for ipv6 machines (#3563)
• Support histogram buckets (#3391)
• Use generic domain for community checkpoint sync example (#3560)
• fix description for BALANCES_CACHE_MISSES metric (#3545)
• Add community checkpoint sync endpoints to book (#3558)
• Pin mev rs deps (#3557)
• Remove strict fee recipient (#3552)
• remove strict fee recipient docs (#3551)
• Add flag 'log-color' preserving color of log redirected to file. (#3538)
• Update graffiti.md (#3537)
• Configurable monitoring endpoint frequency (#3530)
• Builder profit threshold flag (#3534)
• Fixing a few typos / documentation (#3531)
• Strict count unrealized (#3522)
• Add timeout for --checkpoint-sync-url (#3521)
• Fix attestation performance API InvalidValidatorIndex error (#3503)
• Subscribe to subnets only when needed (#3419)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.1.2-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.1.2-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.1.2-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.1.2	sigp/lighthouse