Thread by @TimBeiko

And I'm back for @ethereum #AllCoreDevs!

Our first post-merge call starts in ~40 minutes. See the agenda here: github.com/ethereum/pm/is… 📜

You can watch along on YouTube: 📺

https://twitter.com/TimBeiko/status/1560360677692149760

Will recap on Twitter later, but recommend watching this one live if you care about how The Merge went in detail 🐼!

Ok, we just wrapped up the call! Here we go for the recap 👇

The call started with congratulations 😄🎊 The Merge really went as smoothly as we could have hoped for, and it's all due to teams' hard work over the past years to get us here 💪

Client teams then gave more granular updates: @nethermindeth saw no issues on any of their nodes, and also already spun up new ones which got in sync just fine 👌

@HyperledgerBesu saw a few minor issues: on the merge block itself, a Bonsai Trie issue caused some users to have to restart their node. They are also seeing minor attestation-related and timeout issues which the team is currently looking into, but overall looking very good ✅

@HyperledgerBesu @ErigonEth nodes went through the transition smoothly, but they've observed sporadic sync-related issues post-merge when restarting/re-syncing nodes. The team is digging into it 👀

@HyperledgerBesu @ErigonEth For @go_ethereum, The Merge was, in @peter_szilagyi's words "surprisingly boring" - no issues found, and things were stable enough for the team to spend The Merge following the memes across Twitter & Reddit 😄

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi A few CL teams were on the call too, and didn't report issues - everything looking good on that front too ✨

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi .@MicahZoltu was the only one on the call who ran into serious issues because, while he spent the past few weeks helping users to update their nodes, he simply forgot to update his 😂 All worked fine after he did!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu In terms of chain health, @parithosh_j reported that we're hovering around 98-99% participation rate, which is <1% away from the pre-merge rate, and that we're seeing full blocks on the network - everything working as expected!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j We did see a slight participation rate drop right at The Merge (to ~95%), but it's mostly recovered now. We still haven't had the time to investigate what caused that drop and where the remaining <1% drop is coming from. We should know 🔜!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j The Geth team also gave an update on the PoW side of the fork. So far, we've seen ~25 blocks mined, all using pre-merge Geth releases. The merge geth release was, obviously, optional for miners, but if used, would stop mining once TTD is hit.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j The latest PoW block was 2h old at the time of the call, and we've seen the amount of mined blocks drop significantly because they are no longer being propagated on the network.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j Given that the near totality of nodes upgraded to PoS, and that these nodes stopped gossiping PoW blocks once the network had finalized, there just aren't many nodes who will gossip PoW blocks anymore. Once finality hit, we saw a decrease of PoW blocks mined.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j It's worth noting that there is a rational and non-adversarial reason for miners to keep mining past the first TTD block until finalization: if they mine a valid TTD block, there's still a chance a validator would pick theirs over another one, as it's effectively an uncle block.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j This led us to discuss the Fork ID of the chain. We discussed this on a recent ACD - linking here for context:

https://twitter.com/TimBeiko/status/1545476811458371584

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j In short, each network upgrade, nodes update their Fork ID to choose which peer to connect to. If a peer is on a different fork, you then disconnect them. This is specified in EIP-2124, and works using the fork block number.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j Because The Merge wasn't triggered by a block height, but by the TTD instead, this didn't happen automatically, meaning that pre and post-merge nodes can be peered to each other, rather than fully partitioned at the p2p level.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j To fix this, we can simply specify a block at which to update the Fork ID for merged nodes, as we recently did for Sepolia: blog.ethereum.org/2022/08/03/sep…

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j We discussed if/when we should do that for mainnet (and Goerli, which hasn't had this happen either) and the rough consensus was that it can likely wait until Shanghai.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j Discriminating like this at the p2p layer is most useful if there are many nodes on conflicting forks, but in this case, virtually all nodes have run through The Merge, so it's not looking urgent.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j Given this, and the risk associated with upgrades generally, we agreed that unless something weird happened at the p2p layer, we could simply couple this change with the Shanghai upgrade, effectively activating this no-op fork, and Shanghai, on the same block.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j Next up, @metachris gave an update on MEV-boost. In short, everything went well on that front too. In order to avoid any potential complications _at_ The Merge, the Flashbots relay had been turned off for some epochs after the transition. When it came on, it worked as expected!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris .@hasufl shared the sentiment on Twitter a few hours ago 😄

https://twitter.com/hasufl/status/1570342735147573248

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl Chris also had a proposal for CL clients to add SSE subscription for newPayload events, but given it was mostly EL folks on the call, we agreed to discuss during next week's CL call. He'll put together a full proposal by then!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl We then went over a couple more merge related questions/concerns from teams - I'll summarize here, but worth looking at the recording if you want the nuance around any of these issues.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl First, @adietrichs brought up the concern around ExecutionPayload storage duplication. In short, the EL and CL currently both store these by default, adding "useless" overhead to the storage requirements of running a node.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs CL teams are very aware of this and are in various stages of mitigating the issue already. Prysm, for example, already has a fix for it that's available, and which they expect to make the default soon, after some final testing has been done. Lighthouse has a similar feature.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs That said, these implementations rely on the eth_* APIs rather than the Engine API and it could be better to have all client teams implement a more standardized version of this via the Engine API. @jcksie had a PR about this a while back: github.com/ethereum/execu…

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie CL folks on the call agreed to revisit this PR in the next week or so and discuss further on their next call.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie Second, we discussed @vorot93's suggestion of replacing the TTD by the actual merge block in clients.

While at a first glance, this seemed like it could simplify things, there was some concern about how this could have ripple effects across multiple parts of clients.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 As discussed previously, moving to a fork block would affect the Fork ID (or require custom code to have it _not_ affect it), and also impact how the EL/CL first authenticate each other over the Engine API. It seemed like it could be quite the can of worms 🪱

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 That said, it's worth exploring further, and possibly prototyping in a client such as @AkulaApp!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp This led to some conversations about if and how this initial handshake between EL & CL clients should change, and, if so, how we should integrate this in the Engine API. No strong conclusion here, but you can watch the livestream or monitor the repo for more details!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp Lastly, we briefly touched on if/how we should version the Engine API. As we're discussing all these potential additions, some optional, some breaking, etc. it's clear there's a case to be made for versioning.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp That said, @peter_szilagyi was skeptical, mostly based on his experience with the eth gossip protocol on the EL. With many clients, if you need to track the versions that each of them supports or not, you quickly add a _lot_ of complexity. Even more so in an EL+CL world!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp That said, perhaps something like "temporary versioning" could make sense, e.g. having the "live" version and the "next upgrade" one. Again, no strong conclusion here, and we'll discuss further on the Engine API repo: github.com/ethereum/execu…

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp And that was it for Merge-related stuff!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp Next up, we discussed how to approach Shanghai planning. In short, there's a _lot_ that's being considered for the upgrade, but, also, we just shipped The Merge and people deserve a bit of breathing room 😅

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp My proposal to make progress towards Shanghai but also give everyone a well deserved break is to have a month or so without ACD/CL calls, but try and have async conversations about candidate EIPs for Shanghai.

I shared the full thing on @EthMagicians: ethereum-magicians.org/t/shanghai-cor…

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians TL;DR: EIP Champions can use the ethereum-magicians.org/tag/shanghai-c… tag to signal they'd like their EIP considered, client teams + researchers can use this list to dive into the proposals and discuss their technical soundness, but we *don't* decide on the upgrade scope for now.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians This allows us to make progress on specific EIPs, and get to a good proposal set, in an async way, but people don't have to monitor things super closely and won't, say, leave for two weeks and come back to a completely different upgrade scope than before leaving 😅

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians Of course, this space runs very high on energy and some folks are very eager to keep things moving forward, so even though there won't be ACD/CL calls, we'll have optional voice hangouts in the R&D discord at 14:00 UTC on Thursdays during this time.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians Additionally, during this 'break', we also have @EFDevcon happening!

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians @EFDevcon .@AnettRolikova and me are putting together a F2F @EthMagicians gathering for the conference, where client teams, researchers and the community can have a merge retro, discuss the Ethereum roadmap, as well as specific EIPs 🧙‍♀️

Info here: ethereum-magicians.org/t/ethereum-mag… 😄

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians @EFDevcon @AnettRolikova Finally, @peter_szilagyi highlighted that with The Merge done, there are now many opportunities to simplify Ethereum clients (e.g. all the PoW related stuff can be removed!).

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians @EFDevcon @AnettRolikova So, when thinking of Shanghai, we should consider how we can simplify codebases and not just pile on even more complexity ASAP.

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians @EFDevcon @AnettRolikova So, to recap, in the next month or so we'll have:

- CL call next week (Sept 22, 14:00 UTC)
- Devcon (Oct 11-14), with the @EthMagicians session: ethereum-magicians.org/t/ethereum-mag…
- ACD again, on Oct 27, 14:00 UTC

and optional discord hangs Thursdays @ 14:00 UTC in between 😄

@HyperledgerBesu @ErigonEth @go_ethereum @peter_szilagyi @MicahZoltu @parithosh_j @metachris @hasufl @adietrichs @jcksie @vorot93 @AkulaApp @EthMagicians @EFDevcon @AnettRolikova And that was it! Again, *huge* congrats to everyone involved in The Merge 👏

We did it, Ethereum runs PoS 🐼

Shanghai, we'll see you soon 👀