Thread by @TimBeiko

We wrapped up another @ethereum #AllCoreDevs today, exclusively filled with Merge content ⛓

Agenda: github.com/ethereum/pm/is…
Stream:

Recap below 👀

https://twitter.com/TimBeiko/status/1515084519811928065

@ethereum First on the call, @parithosh_j gave a recap of the two shadow forks we did last week: one on Goerli and one on Mainnet. The Goerli one happened on Tuesday and some client teams had issues, that said, most were fixed by Saturday's mainnet shadow fork 💪🏻

@ethereum @parithosh_j The mainnet work went relatively well, where we didn't see major issues and the network promptly finalized post-merge 😁 Pari had a tweetstorm about it last week:

https://twitter.com/parithosh_j/status/1517816757884538881

@ethereum @parithosh_j And so did Adrian from the @Teku_ConsenSys team:

https://twitter.com/ajsutton/status/1517869283883331586

@ethereum @parithosh_j @Teku_ConsenSys As these threads highlight, though, there were some issues discovered around processing a large amount of deposits on the beacon chain in a short amount of time, which led to delays in block proposals on the network.

@ethereum @parithosh_j @Teku_ConsenSys Additionally, we found some proposal-related issues in a handful of EL <> CL combos. Most of them have been identified or fixed, but Erigon still has some open ones.

Overall, we're currently sitting a 96% participation rate on the shadow fork.

@ethereum @parithosh_j @Teku_ConsenSys Then, we spent some time on the call going over the specific issues we're seeing, and possible mitigations/test scenarios.

@ethereum @parithosh_j @Teku_ConsenSys First, @ErigonEth mentioned users had reported some sync issues which they are looking into, and that they are prioritizing fixing failing Hive tests (see: hivetests2.ethdevops.io)

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth Then, @peter_szilagyi highlighted that all our shadow forks rely on clients which are fully sync'ed, and that it would be valuable to test client combinations where either the EL or CL isn't fully sync'ed when The Merge happens.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden mentioned he had done some of this manually on Geth during shadow forks, but agreed it would be nice to see more of it.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden We agreed to have clients pay special attention to this during next week's mainnet shadow fork. Sync, and its various edge cases, is really the major thing most client teams are focused on making robust right now.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden While it is reasonable to expect most of the network to be in sync when The Merge happens, we want to make sure we are at least aware of the potential failure modes and solve for as many edge cases as possible!

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden We then had a conversation about how an EL node sync should work in the case where it is missing its CL counterpart on a network which transitions to PoS.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden Post-merge, it's easy for clients to update their configs to be aware that The Merge has happened, but for the client release which actually goes through The Merge, this isn't an option.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden The two options are basically to either loudly fail, or for the node to try and recover. While the former seems obvious, there are cases where, when setting up both an EL and CL node, it could fail even though it shouldn't.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden We didn't make an unanimous decision here and decided to have various teams handle this as they think is best. In addition to this, we'll make sure to keep communicating exactly how node operators, stakers, and infra providers should configure their nodes for The Merge.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden ICYMI, the Kiln announcement post covers a lot of this: blog.ethereum.org/2022/03/14/kil…

🚨 IF YOU ARE A NODE OPERATOR / STAKER / INFRA PROVIDER AND HAVEN'T LOOKED AT THIS YET, YOU DEFINITELY SHOULD DO THIS ASAP 🚨

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden In addition to this, the Geth team has started working on new debugging APIs to help triage issues that come up during shadow forks. They encouraged other client teams to also support them to make cross-client diagnosis easier. github.com/ethereum/go-et…

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden We then discussed how we could potentially go about automating shadow forks. We've been working with @KurtosisTech for merge testing and pushing the team to their limits 😛

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech We agreed that between this and their other project, creating partitions and other disruptions in merge networks, it made more sense to have them work on the latter.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech After this, @mkalinin2 gave an update on the latestValidHash issue we discussed on the previous call

https://twitter.com/TimBeiko/status/1515069928922968069

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 In short, after some f2f discussions in Amsterdam, we agreed to leave the Engine API spec as is, and add additional test cases to ensure the correct behaviour across various edge cases.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 With this closed, Mikhail also highlighted another spec edge case, where, if the first post-merge block is INVALID as per the Engine API, it's unclear what the CL clients should do. github.com/ethereum/execu…

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 In short, the behavior for other blocks is to then go back in history to find the latest valid block in that fork. But, CLs don't have pre-merge blocks as part of their block tree. There are a few options for how to address this which Mikhail details in the issue.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 No one had had the time to properly review it (it got created 9h ago!), but there wasn't a sense than any option would be catastrophic. We'll spend time looking into this in the next week or two.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 Next up, we discussed another outstanding issue about how to label new JSON RPC parameters post-merge: github.com/ethereum/execu…

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 There was some back and forth on the call about if and how we should use safe/unsafe/latest, but I didn't want us to spend too much time on the topic so invited folks to discuss it async over the next couple weeks. The livestream has the pros/cons of each 😁

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 That said, it *is* important that we agree to *something* soon, as this is behaviour we'll want standardized across clients before we move to forking public testnets. So, while I don't have a strong opinion about the outcome, I feel strongly about getting to *an* outcome 😅

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 After this, @lightclients had some design questions related to mev-boost, which is Flashbots' post-merge product to communicate with Ethereum clients.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients In MEV boost, block builders send entire blocks (rather than transaction bundles) to validators. One of the parameters that's set as part of a block is the gas limit for it, where miners/validators can raise/lower it from the previous block's limit.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients So the core question is whether you want validators/block proposers to impose a specific gas limit to the block builder, or allow the builder to set their own limit. After some back and forth, we agreed that the validators/proposers should set it, not builders.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients In other words, there will now be an extra constraint in mev-boost, where a block builder needs to build a block which is valid *and respects the validators' chosen gas limit*.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients Next up, we discussed the future of testnets after The Merge. There was some discussion about this at the @EthMagicians gathering last week, summarized here: ethereum-magicians.org/t/og-council-p…

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians One thing that became clear there is that the community hates the idea of deprecating testnets! So much so, that some orgs have stepped up to offer to maintain them post-merge.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians So, from the client development PoV, we will continue forward with having Goerli & Sepolia (soon on @_anishagnihotri's faucet!) as the long-term maintained testnets, and running Ropsten through the merge with the intention of deprecating it. That will provide enough test data.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri That said, if an org wants to maintain Rinkeby/Ropsten, we'll try and make it work. I'm already chatting with some folks about how this could work, expect more updates on this soon.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri Then, I moved to the tricky part of the call: asking client teams their thoughts about when we should start considering forking testnets 😅👀

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri Different people brought up different criteria, here they are:

1. We want to see successful mainnet shadow forks which go smoothly. The general feeling was that our last one was _just below_ the bar we'd like to see, and if we can be a bit smoother, 1-2 more should suffice.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri 2. We want to see all clients passing the vast majority of hive + other test suites. We're making good progress here, but not all client teams have had the bandwidth to add support + review all failures yet.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri It's worth noting that some failures can happen because the test set up is wrong, but we want to be sure of that!

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri 3. More fuzzing and edge case testing. Ideally, we'd like to fuzz each client pair to find issues, and test things like the out-of-sync-during merge scenarios that @peter_szilagyi brought up early on.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri And, finally, we discussed how all of this fits in with the difficulty bomb 💣

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush came on the call to present his projections of the bomb: ethresear.ch/t/blocks-per-w…

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush He argued that when the bomb starts to show, it shows quickly, and so we want to be ready to react.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush My stance was that, ideally, we should wait as long as we possibly can before committing to a decision about the bomb, because *if* there is a path to the merge where we avoid a bomb delay, we should try and do it to not add the delay caused by another fork for the bomb.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush There were some concerns from @nethermindeth that this will be hard to predict because the hashrate may drop, and that when the bomb goes off, the experience for end users will be bad. Also, moving the bomb is fairly easy, as we've done it multiple times before.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu also suggested that, if we are so stressed out about the bomb, we could also simply remove it to give us the space to focus on delivering the merge as well as possible, without worrying about dates.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu There was some support for this idea across client teams. That said, @dannyryan also pointed out that having the bomb does act as a mechanism to prevent scam PoW forks from emerging right at The Merge.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan In short, if a PoW fork needs to push back the bomb, it shows that the team working on it at least has _some_ technical know-how and that they can convince their users to download their software, rather than stay on the default chain.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan The bomb increases in impact every ~2 weeks, which means we can review its progress on each call and assess what the right next step is. There are probably ~2 more periods before block times are >1 sec longer than right now.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan We were at the end of the call by then, so we agreed to re-assess in two weeks, and until then focus on:
1. Having two more mainnet shadow forks 👻
2. Getting clients supporting and passing hive test suites ✅
3. Fixing the outstanding bugs from past shadow forks 🐞

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan And that was it! Next ACD is set for May 13 14:00 UTC - see you then 👋🏻

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan One more important note on naming specific clients' progress status/outstanding issues: the main reason I do this is to try and share discussions which, while they happen mostly publicly, rarely surface to the broader community.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan EVERY. SINGLE. TEAM. has been working insanely hard on making sure The Merge is safe, and that their software supports it. It's definitely not the case that a single team is holding the process behind, for ex:

https://twitter.com/vdWijden/status/1520113035901378561

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan Having multiple teams working on this has made the design better, led to issues being found which we would have missed, and generally is the only way to get so many smart + talented folks to work on Ethereum.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan We've found issues in basically every client: one week it sucks for geth, the other week nimbus has an issue, then it's nethermind, erigon, lighthouse, prysm, etc.

I think it's good to share with the community what those issues _are_ (and how we fix them) 😁

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan But it can quickly become unhealthy if the team which finds the last issue (which other clients may also have and simply not hit!) somehow gets blamed for delaying/slowing down the process.

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan Having a client crash or hit a weird bug late into the process isn't "delaying the merge", it's making sure that Ethereum doesn't go down during the upgrade, which is the core thing we are all trying to achieve 😄

@ethereum @parithosh_j @Teku_ConsenSys @ErigonEth @peter_szilagyi @vdWijden @KurtosisTech @mkalinin2 @lightclients @EthMagicians @_anishagnihotri @tjayrush @nethermindeth @MicahZoltu @dannyryan Oh, one last thing: @christine_dkim again had a recap out before me 😄 Recommended for an additional perspective on the call !

https://mobile.twitter.com/christine_dkim/status/1520087529139019779