Thread by @TimBeiko

We just wrapped up a *very* eventful @ethereum #AllCoreDevs

Agenda: github.com/ethereum/pm/is…
Stream:

Will recap here, but IMO if you only listen to a handful of calls per year, this one should be on your list 🎧

https://twitter.com/TimBeiko/status/1530284959365492737

@ethereum First on the call, we discussed the Ropsten Merge, which happened earlier this week. @dannyryan gave an update, based in part on @parithosh_j's analysis from earlier this week:

https://twitter.com/parithosh_j/status/1534653280147718148

@ethereum @dannyryan @parithosh_j Pre-merge, some CL clients had issues tracking deposits to the beacon chain. This was because they made specific assumptions around block times which did not hold under Ropsten's chaotic block times due to its variable hash rate as a testnet.

@ethereum @dannyryan @parithosh_j These issues were all fixed before hitting TTD and deposits were all processed normally pre-merge.

@ethereum @dannyryan @parithosh_j After The Merge on Ropsten, we saw a ~14% participation drop on the network. Danny has tweeted about this:

https://twitter.com/dannyryan/status/1534928766022979584

@ethereum @dannyryan @parithosh_j With the tweaks mentioned in the above tweet, participation rate went back up to ~99%, which is similar to mainnet. It's now a bit lower as teams are testing fixes and running other tests on live validators.

@ethereum @dannyryan @parithosh_j We've also started running @vdWijden's fuzzer on Ropsten to make sure we have a good transactional load on the network.

@ethereum @dannyryan @parithosh_j @vdWijden We dove into the client issues in more details on the call. Again, strongly recommend watching the livestream to get all the nuance.

@ethereum @dannyryan @parithosh_j @vdWijden For @nethermindeth the concurrency issue occurred because, at TTD, some nodes would receive new blocks both from the EL p2p network _and_ their CL client. This was made worse on Ropsten because of the high uncle block rate due to the bad mining conditions.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth The team is currently working on a fix for the issue 🛠

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth The @HyperledgerBesu <> @ethnimbus issue had to do with the interaction of websockets, JWT tokens and missing eth_* API endpoints exposed at a specific port.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus The Nimbus team shared a workaround with Besu which is working now, but it requires disabling JWT. The Besu team is currently working on a proper fix for the issue.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus In addition to issues leading to lower validator participation, there was also an issue where @ErigonEth was producing blocks with 0 transactions in it.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth This specific bug was fixed, but the "deeper" cause is that Erigon does not support PoW mining. This means their code for block creation isn't s robust as other parts of the client and they are now focusing heavily on that part of the codebase.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth Next up, we discussed an issue around CLs asking ELs for blocks before these have had time to create a block. There was a lot of talk about the intricacies of this interaction on the call (@ajsutton even showed up!) so, again, recommend watching the livestream for the details!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton In short, some CLs sometime ask ELs for a block before these have had a time to create one. Per the spec, the correct behaviour is for clients to return an empty block. This obviously isn't ideal because the EL can usually create _some_ block pretty quickly.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton Because of this, Geth & Nethermind have implemented a workaround where, if this happens, they simply create a block "ASAP" and send that back. This isn't great, because it doesn't fix the root cause of CLs sending asking for the block too soon.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton Again, most CLs have this fixed already, and others are focusing on it now. Once this is fixed, ELs will likely revert this functionality to instead follow the spec. In short, when they get asked for a block, they'll stop iterating on its construction, seal it, and send it back!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton Next up, @ralexstokes had a change to the Engine API he wanted feedback on: github.com/ethereum/build…

In short, this would allow CLs to pass a specific gas limit as part of Engine API calls, which would be useful for external block builders.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes To unpack that a bit more: post-merge, validators control the gas limit on Ethereum, like miners do today. This parameter is set as part of the EL, when producing a block. Each time a block is chosen, you can raise/lower the gas limit by 1/1024th, or leave it as is.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes If you assume that a staker is the one producing their block, then the staker passes their desired gas limit to their EL, and that's it! Things get more complex in a world where the staker may not be building the block themselves!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes In practice, this is likely due to MEV. Validators who want to benefit from MEV fees need to run additional software to receive these blocks from searchers/relays.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes (side note: the entire MEV design around the merge is out of scope, but @lightclients had a great thread about it recently

https://twitter.com/lightclients/status/1533915131536605185

)

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients So, for external block builders to be able to follow validators' gas limit preferences, it would be useful to have them be passed explicitly as part of Engine API calls. This is what the proposed change does :-)

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients We agreed on the call that this should _not_ come as a change to the current API, but rather as a separate version for it. There were also questions about whether other info should be passed to builders, such as the `extraData` field.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Everyone seems to agree this is useful, but there are still some details to iron out, which we'll keep doing over the coming weeks!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients After this, we discussed the launch of the Sepolia Beacon Chain. With Ropsten now merged, Sepolia also needs a PoS chain to merge with 😁 The configs have been mostly finalized over the past week: github.com/ethereum/pm/is…

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Similarly to Clique, the validator set on Sepolia will be permissionned. The Beacon Chain will go live in 10 days, and quickly run through Altair + Bellatrix with an artificially high TTD. When we are ready to merge Sepolia, we will perform a TTD override, like for Ropsten.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients After this, I asked the various EL teams what they'd like to see before being confident in moving forward with testnets. Once more: there was plenty shared on the call, these are just rough highlights, please watch the livestream 📺!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Nethermind went first: for them, it was seeing hive/kurtosis tests all passing, block proposal issues fixed, and obviously having their current concurrency issue fully resolved. They said they'd want at least 1/2 of the testnets to have code which is basically final.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Erigon echo'ed passing tests, saying they still have a lot of them to fix themselves. Beyond that, they reiterated wanting to solidify their block building code, as well as resolving an issue related to returning the `latest` block in JSON RPC post-merge.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients That last bit has to do as to whether the `latest` block should be updated only when a CL updates its fork choice, or if any new block sent which is deemed by the EL to be on the canonical chain should be `latest`.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients The spec uses the former, but some EL implementation, especially Erigon, have a database layout which makes this hard, given they only keep a single "current" state.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Besu concurred with everything related to tests, with an emphasis on sync. We spent some time on the call discussing what assumptions ELs should make when they don't have the head of the chain from their CLs, and how to communicate that to users.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Geth agreed with other teams, and also said it was missing a JSON RPC flag for the "safe" block.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Teku, Prysm and Lighthouse all had representatives on the call as well. They all seemed pretty ready to move to the next testnets, each having small outstanding issues, but nothing they see as blockers.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Last but not least, Flashbots also had someone on the call. There were some conversations about the readiness of mev-boost and if/how that should be a blocker. Staking pools seem concerned about monitoring of MEV, and the FB team about user education for solo stakers.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients There wasn't a strong consensus around whether this was a blocker or not, but obviously work will continue on implementation, testing and education around mev-boost

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients Lastly, with little time left, we had two big topics left: how to order the next testnet deployments and if/how to delay the difficulty bomb. This was the messiest part of the call because of the many intertwined topics, so, once more, the livestream has 100x the nuance 📺

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients First, w.r.t. testnet deployments, it's worth noting how Goerli & Sepolia differ. Goerli is a large testnet, with a ton of activity and where stakers can run through The Merge (with the Prater Beacon Chain).

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients OTOH, Sepolia is a new testnet, with little activity and a permissioned beacon chain. Goerli will give us more data on many fronts, and Sepolia will be easier to transition.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients We didn't explicitly agree to this on the call, but the consensus seems to have shifted towards doing Sepolia before Goerli, once the issues we have on Ropsten have been resolved, so that Goerli can be merged with code that is as close to mainnet's as possible.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients The Sepolia Beacon Chain is being launched with this assumption. Note that we could still change it if we had a good reason to, and two weeks ago the plan was to do the opposite. We didn't make a call because we segwayed into the difficulty bomb 💣!

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients The bomb is being felt on the network, and, in true bomb fashion, it appeared quicker than predicted 😅 Block times are ~14s and the Arrow Glacier EIP (authored by yours truly) predicted "a ~0.1 second delay to block time by June 2022 and a ~0.5 second delay by July 2022."

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients So, @tkstanczak, who has been advocating for us to delay the bomb for some time now, drafted an EIP and proposed it on the call: eips.ethereum.org/EIPS/eip-5133

The EIP proposes to push back the bomb ~500,000 blocks, or 2.5 months.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak So this means that block times would be back to where they are today (~14s, roughly 1s delay) by late August. The bomb's impact goes up exponentially every 100,000 blocks, so it ramps up quickly after that, as seen in previous periods.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak This was contentious, to say the least. Again, recommend watching the livestream for the back and forth. How much pressure is the right amount to put client teams under, the impact of block times on users, and of PoW on the environment all came up 😅

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak In short, we agreed to the bomb delay. We were already over time, and want to be sure that we sanity check all the numbers before selecting an exact delay and deployment time, but we are aiming for a ~2 month delay, and for the upgrade to go live late June.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak All EL teams were in agreement that doing this soon would make things better for users, and that having a short delay would mean we keep a strong focus on The Merge.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak Again, we were already over time here, so we want to sanity check the numbers async, which should happen over the next few days. Realistically, we'll have client releases and a proper announcement out by the next of next week.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak I'll wrap up by quoting @vdWijden's passionate end of call speech, saying that the bomb delay is a "five line change" for clients and is easy to implement. He feels strongly this doesn't slow down team's work on The Merge, and @tkstanczak has also stated the same before.

@ethereum @dannyryan @parithosh_j @vdWijden @nethermindeth @HyperledgerBesu @ethnimbus @ErigonEth @ajsutton @ralexstokes @lightclients @tkstanczak And that was it! We had to scrap everything else on the agenda to get through this. Hopefully, we can get to them on the next call, scheduled for June 24, 14:00 UTC. See you then!