Thread by @lightclients

Ethereum clients currently store 275 GB of historical data that is unnecessary to validate the chain. That number is growing at a rate of around 140 GB per year. EIP-4444 proposes clients prune data older than 1 y/o.

So why don't we just prune the data already?

To understand why the data has yet to be pruned and why this is even a discussion, it's important to understand how historical data is used today. There are two main categories of usage: syncing and user requests over JSON-RPC.

In the world of syncing, there are two main approaches:
- Full sync - from genesis, download every block & execute it until tip of chain is reached.
- State sync - many schemes here, but essentially header sync using PoW checks and then download the state for the latest block.

In both cases, clients request historical data over the p2p network to develop their view of the chain. The trust model has generally been to trust the genesis state and verify everything else - either fully verifying or light verifying with PoW checks.

Proof-of-stake changes this. Because it's susceptible to long range attacks, we must rely on a "Weak Subjectivity (WS) Checkpoint". This is essentially a block in the canonical chain that we trust with the same level of trust conveyed onto the genesis block in PoW.

The WS checkpoint allows clients to skip the bootstrapping step of requesting historical data over the p2p network. Of course, they'll still need to sync the historical data after the checkpoint - therefore the checkpoint should always be before the pruning boundary.

This sounds like a regression in security. Before, we had one hash from 13 July 2015 to verify. Now, we have a moving WS checkpoints. But in reality, we've been relying on weak subjectivity all along.

When is the last time you verified the code diff between client releases? Most people don't have the technical background to do this. So, every time you update your client, you're relying on the client team to faithfully implement the Ethereum protocol.

Fortunately there are a lot of eyes on software like go-ethereum. It takes just one whistleblower to show a malicious commit in the code. Similarly, it only takes one whistleblower to point out that a client is shipping with a malicious WS checkpoint.

In fact, it's much easier to verify a client ships with a correct WS checkpoint than it is to ensure the code correctly executes the protocol.

So, from a security perspective there is really no regression.

That covers syncing - the other main category of usage that historical data is needed for is serving user requests.

There are two types of data that users can request:

- current data, e.g. the value of a storage slot, an account balance, the latest block number, etc
- historical data, e.g. the value of a storage slot at block N, the header of block N, a transaction receipt, etc

Current data will continue to be accessible, however with EIP-4444 the historical data may not be depending on how long ago it is from.

The main consumer of historical data are dapp devs. Many dapps populate their databases with historical information to serve to their users via their frontends. For them, it's important to be able to iterate through all txs and logs.

There are various ways to support this use case - currently the favored approach is a client release multiplexer that executes certain block ranges on releases that support the range. For example, geth vA may support up to block 10m and geth vB supports 10m+.

The multiplexer would execute blocks 0-10m with geth vA, output the state db and import it into geth vB, then continue with blocks 10m+. JSON-RPC requests would be directed to the client that has the appropriate information to respond.

However, if the historical blocks are no longer available on the p2p network - where do they come from?

It's expected that many large, trusted institutions will provide mirrors to this data. Since the data is static, it's easy to agree on their hash and verify. 1-of-N trust.

The new standard will be to *not* store historical data and run a client multiplexer. This means the standard footprint of Ethereum clients will be reduced by 275 GB - however there is final issue to mention.

Currently the Ethereum JSON-RPC responds with an empty response when a requested piece of data does not exists. Assuming the client is not syncing, this can be accepted as "this piece of data does not exist in the canonical chain or recent fork".

Once clients begin pruning old data, this invariant breaks. When a user requests a certain tx receipt, the client won't know if the receipt was pruned or just never existed.

Currently, the expectation is that the RPC will return an empty response in both cases.

I'm curious to get feedback on this approach. How do consumers of the JSON-RPC feel about this? How often do you access historical data over 1 y/o? An alternative (albeit heavier) would be to maintain an index of pruned hashes so that more context could be returned to the user.

The 275 GB number was ascertained from the output of `geth db inspect`. Here is a screenshot:

The official EIP-4444 (pronounced EIP four 4s btw) specification can be found here: eips.ethereum.org/EIPS/eip-4444