Prysm root calculation error causes Ethereum mainnet chain failure

According to PANews reports, the Prysm team has released a detailed analysis of the technical issue that occurred during mainnet operation on December 14th. The core of this incident was that beacon nodes received authentication from nodes that were not synchronized with the network, leading to unexpected errors in the root calculation process. This issue went beyond a simple client error and had widespread effects on the entire network.

Authentication from Unsynchronized Nodes Triggers Chain Reaction in Root Calculation

The problem was triggered when Prysm beacon nodes received authentication referencing the previous epoch’s block root from unsynchronized nodes. To verify this, Prysm attempted to reconstruct the beacon state to be compatible with the chain state recognized by the unsynchronized nodes. However, this root calculation reconstruction consumed more computational resources than expected.

As a result, blocks from the previous epoch were processed redundantly, and epoch transition operations, including root calculations, caused excessive load. In fact, this vulnerability was already included in Prysm PR 15965 and was deployed on the testnet a month before the incident, but it was not detected at that time.

Excessive Resource Consumption Due to Root Calculation Overload, Validator Responses Become Unavailable

During the Fusaka session on the Ethereum mainnet on December 4th, almost all Prysm nodes experienced processing delays during the root calculation process. When handling certain proof data, computing resources rapidly depleted, causing nodes to be unable to respond promptly to validator requests.

This resource shortage led to a chain of block and proof losses. The issue persisted across 42 epochs from epoch 411439 to 411480, with 248 blocks missing out of 1,344 slots, resulting in a severe omission rate of approximately 18.5%.

Network Participation Dropped to 75%, Validators Lost 382 ETH

The impact of this incident went beyond simple block loss, threatening the overall stability of the network. Network participation rate once plummeted to 75%, indicating that the Ethereum consensus mechanism was not functioning properly.

Validators lost about 382 ETH in rewards due to block and proof losses. This not only caused financial damage to individual validators but also undermined the overall trustworthiness of the network.

Lessons from Root Calculation Error, Reassessing Network Resilience

Interestingly, Vitalik Buterin has previously mentioned that occasional “loss of finality” in Ethereum does not pose a serious risk. However, this Prysm root calculation error incident clearly demonstrated how a single client flaw can impact the entire network. Moving forward, diversity among clients and robustness of core operations like root calculation are expected to become even more critical.