Thread by @TimBeiko

We wrapped up another @ethereum #allcoredevs earlier today - covered the latest on Merge testing, testnet plans, and a handful of EIPs 😁

Agenda: github.com/ethereum/pm/is…
Stream:

Recap below ⬇️

https://twitter.com/TimBeiko/status/1509974571025256448

@ethereum Again, @christine_dkim also shared a recap, which I strongly recommend 👀

https://twitter.com/christine_dkim/status/1515013002105442312

@ethereum @christine_dkim First on the call, we went over the recent mainnet shadow fork. For those of you not familiar with shadow forking, I wrote a short overview here: github.com/timbeiko/eth-r…

@ethereum @christine_dkim While the mainnet shadow fork was a "success", in that it managed to finalize valid blocks post-merge, several clients ran into issues during it. We covered some of these on the call.

@ethereum @christine_dkim First, @nethermindeth had some sync issues around the transition. These weren't new, they had been discovered during the Goerli fork, but the team wanted to make sure they didn't miss a single test run so ran with it despite not having fully patched the issue yet.

@ethereum @christine_dkim @nethermindeth Then, @HyperledgerBesu also hit a few issues. The first was a simple config issue which was quickly solved. The second had to do with fast sync: when the pivot block moved from a pre to post-merge block, the post-merge rules weren't automatically applied. That's being fixed too.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu The third issue, which they still haven't found the root cause of, was caused a few blocks after the merge, where the client hit a receipt root mismatch, meaning that it came to a different end state after executing a transaction. They are still investigating what went wrong here

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum, like Nethermind, ran through the fork with an old issue still unresolved. Some nodes during the goerli shadow fork ended up with a corrupted state snapshot, and the team still hasn't found the root cause.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum Finally, @ErigonEth got through the fork smoothly, and even picked up a bug in some CL clients, which were making calls to unexpected JSON RPC endpoints via the Engine API

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth On the CL side, Prysm was on the call. They found a few edge cases during the mainnet fork, none of which were consensus-breaking. Beyond these fixes, another thing they are working on is improving the UX of running Prysm along with an EL.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth They've seen many users struggle with this in their discord recently, and want to make sure that their docs, and clients, are clear about how to run both layers in tandem.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth We have two shadow forks planned for next week: one of Goerli scheduled for next Tuesday, and another mainnet one scheduled for Saturday 👀

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth Next up on the call, we discussed some of @mkalinin2's latest analyses around latestValidHash. We discussed this on the last call (

https://twitter.com/TimBeiko/status/1509960329396047878

) and came to the conclusion that we could perhaps be more lax in the EL client behavior here.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Thinking about it more, Mikhail may have found a security issue caused by relaxing this behavior. He wrote a doc about it, and walked through it in detail on the livestream. Recommend watching the whole thing! hackmd.io/GDc0maGsQeKfP8…

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 In short, the issue would arise if an attacker could force clients to re-org to a block for which they've discarded the parent's state, because they originally thought it wasn't part of the canonical chain and discarded it.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Clients usually only keep fairly recent state diffs (64-128 blocks), so such an attack which caused them to re-org to a previously-thought-non-canonical chain beyond this could cause them to not be able to eventually re-org to the valid chain.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 There were a few questions about this on the call, but one thing that came out is that an attacker would need to control a very large portion of the stake to launch this attack.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 People still need to digest this, but over the next two weeks we'll look at how easy a mitigation is to implement across clients, and exactly how much stake would be required to cause this, and whether at that stake amount more serious attacks are already possible.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 If worse things are already possible at that % of stake and it's non-trivial for clients to implement a fix for, it may not be worth it to patch this. That said, if one of those things isn't true, we'll want to make sure we are safe against these scenarios and have test for them.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Next on the call, we discussed how we want to approach testnet deployments for The Merge. We had briefly touched on it on the past call, but because we were short on time, never got deep into the discussion.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 The @nethermindeth team shared some thoughts on the agenda ahead of the call:

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 We had a lot of back and forth on the call, and I spoke a fair bit so don't have great notes, but here's my summary of where we ended up - the livestream has all the back and forth that got us there 😁

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 First, given that we've added Shadow Forking to our testing process, which we didn't do for previous Ethereum upgrades, we can assume that once we move to testnets, most/all issues will have been ironed out.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Again, shadow forks are basically testnet/mainnet forks that _actually happen_, but only on a very small number of nodes, controlled by the client + testing teams. We'll want to keep shadow forking until things are very stable.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 At that point, the main thing we want to ensure is that node operators are able to correctly configure their nodes for the upgrade. Nethermind is spot on that this upgrade is more complex for node operators than previous ones.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Ideally, everyone would have already done this on Kiln, as we've been urging folks to do for months (all the info here: blog.ethereum.org/2022/03/14/kil…, still the time to do it!!), but we know in practice that won't be the case 😅

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 So, given that, it's reasonable to give folks a bit more of a heads up before we fork the first testnet, and to expect that things may not go super smoothly given there will be a lot of first time combined EL & CL node operators.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Because we want to target as many existing users as possible, and that there is a larger risk of misconfiguration, we'd rather run this process on a network which will be shut down some time after the merge: Ropsten

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Once Ropsten is stable, we'll then move to forking Goerli, and then Sepolia, in close succession. This way, by the time all three testnets have been forked, Ropsten will have been running post-merge for a fair amount of time.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 This means we can test things like node stability, syncing fresh nodes, etc. on Ropsten in a post-merge context, and make sure things are working as expected. It's a good network to do this on, because it does have a large state & history.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 One thing which is implicit here, is that Rinkeby *won't* be transitioned through The Merge.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 So, if you are using the network, we recommend moving to either Sepolia or Goerli, both of which are expected to be maintained post-merge. Neither Rinkeby nor Ropsten will be shut down overnight, but they will be deprecated 🚨

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 The difference is that Ropsten _will_ run through the merge, and still work as a testnet until it is shut down, while, post-merge, Rinkeby won't actually be a good staging environment for mainnet, as it will be one upgrade behind.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 Finally, two more things we discussed as being important to get done before moving to public testnets:

1. Make sure that all clients only serve the Engine API over an authenticated JSON RPC port. We want to force users to have to configure this before the Ropsten fork.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 2. Run Shadow Forks where more of the validators are controlled by client teams. Up to now, while client teams do run validators in Shadow Forks, the EF devops team (hi @parithosh_j !) has been controlling enough to ensure finality is hit on the networks.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Once things are a bit smoother, it makes sense to distribute validators more broadly to have testing conditions that mirror public networks a bit more.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j So, to summarize: The Path From Here to The Merge 🏞

1. More Shadow Forks, which don't result in client issues, and where client teams control a large % of validators 👻
2. Fork Ropsten: give folks plenty of heads up to configure their node correctly, hope they do it right🤞🏻

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j 3. Fork Goerli & Sepolia: ensure things go smoothly, and that Ropsten has stabilized, new nodes can join, etc. 😌
4. Wait a bit, make sure all testnets are looking good ✨
5. Then, run through the process on mainnet 🎉!

Again, Rinkeby *won't* be run through this transition 🚫

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j That was it for the Merge part of the call! Next up we discussed one of the EIPs currently being considered for Shanghai: EIP-3540, which introduces the EVM Object Format, or EOF

eips.ethereum.org/EIPS/eip-3540

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j TL;DR on EOF from my last AllCoreDevs update (hackmd.io/@timbeiko/acd) 👀

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j The first thing we discussed about this was whether we could use EOF as a way to start removing SELFDESTRUCT.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j This is something we've long wanted to do, and now have a reasonable proposal for (eips.ethereum.org/EIPS/eip-4758), but it's not clear we want to add more EIPs in Shanghai (more on that later!)

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Given that EOF already specifies new execution rules for its contract, it would be easy to simply remove SELFDESTRUCT from it from the start.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j There was some back and forth on the call, and no firm conclusion about next steps, but one idea was that perhaps we could simply not enable that opcode in the first version of EOF, and then subsequently decide to activate it if we think we need it.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j It's not a clear cut positive, because if we replace SELFDESTRUCT by SENDALL (which sends ETH in the contract without removing the code), there may be some legitimate reasons to want to use SENDALL in EOF.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Then, there were some questions about how EOF would interact with Verkle Tries, which are a requirement for Stateless Ethereum, especially with regards to code chunking.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j To unpack all of this: stateless ethereum allows us to scale how much throughput we have at the execution layer, by only requiring most nodes to store a subset of the state rather than the entire state.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j To prove to another node on the network, which may not have all the state either, that a transaction is valid, we therefore need to start sending mini parts of the state, called witnesses, around the network.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j ELI5 version: today, all nodes store everyone's balances and all the smart contract code. In stateless, maybe your node just store your balance, along with the smart contracts you care about (e.g. the NFTs you own).

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j So if you want to send 1 ETH to someone, the other node needs a proof you actually have 1 ETH, so you just the part of the state to your transaction, along with the transaction itself.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j The challenge is that with our current storage structure on Ethereum, Merkle Patricia Tries, these witnesses/proofs/mini-states can be quite big. Think of it like a large christmas tree, and the proof you need to send goes from the tip of it to the bottom.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Verkle tries, instead, are much "flatter". Think of it like a bush (but very wide and triangular shaped... bear with me 😅)

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j So, in this case, the proofs are much smaller because the distance from the top/head of the tree to the base/root/data you want to prove is shorter. This means these proofs are easier to send around the network!

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Verkle Tries require us to change how we group (or "chunk") the state data in Ethereum before we send it around.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j EOF, by introducing a separation between contract code and data, creates new "boundaries" in contracts which didn't exist previously. Therefore, it may cause issues if when we naively group/chunk data, we cross those boundaries.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Luckily, it seems the EOF authors have considered this already, and are generally compatible with the Verkle Tries specification. There are some optimizations that can be done, but they are not must-haves 🎉

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j This was quite the detour, but hopefully did a decent ELI5 version of the livestream. Also recommend this post for more context on Verkle trees: blog.ethereum.org/2021/12/02/ver…

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j If you're still following, there's one more thing we discussed re: EOF: whether there are any incentives to move to it.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j While there won't be any direct gas-reductions to use EOF, it will eventually enable patterns that lead to lower gas usage for common use cases, such as control flow. The team is currently trying to get concrete numbers for, say, Uniswap transactions.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j And that was it for EOF!

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j Next up, we had a quick announcement: on the previous ACD there were several EIPs which asked to be considered for Shanghai. That said, we already have a decent list (see: github.com/ethereum/execu…) and haven't started implementing/testing any of them, as we're still on The Merge.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j So, I asked client teams if they wanted to pause considering new EIPs for Shanghai before we're much further along in the process and the answer was a unanimous "yes", see this from the R&D discord:

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j While we had agreement on that, @adietrichs wanted to get people's opinion on an EIP which we had previously considered bundling along with The Merge or Shanghai: EIP-4396 eips.ethereum.org/EIPS/eip-4396

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs In short, the EIP makes the EIP-1559 base fee dependent on time, rather than simply "the last block". This is helpful to deal with missed slots under PoS, which lead to more transactions accumulating in the pool, without extra elasticity in block sizes.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs While the EIP is definitely a nice-to-have from a UX standpoint, as it makes the base fee less volatile, there isn't strong consensus that it's a must have for Shanghai.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs There are a few security considerations around block proposer DoS attacks and reducing the chain throughput, but there isn't strong consensus that these are issues which we want to address with this EIP.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs We've agreed to keep the EIP updated, and have it in our back pocket if needed, but not commit it to Shanghai unless we see a strong need to do so, post-merge.

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs Finally, @greg_colvin had some more questions about EOF, but we struggled to get into the core of the issue on the call. The conversation continued on the R&D discord, and is likely to continue in Amsterdam next week 🇳🇱

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs @greg_colvin And that was it! Next AllCoreDevs is scheduled for April 29th, 14:00 UTC. Before then, see you at @EFDevconnect 😁!

@ethereum @christine_dkim @nethermindeth @HyperledgerBesu @go_ethereum @ErigonEth @mkalinin2 @parithosh_j @adietrichs @greg_colvin @EFDevconnect Bonus shadow fork-related thread from @nethermindeth's @M25Marek 👀

https://twitter.com/M25Marek/status/1515080026768740355