@ethereum TL;DR of the issue is that the Ropsten TTD was hit before the Beacon Chain for the testnet was live and Bellatrix, the CL fork which activates merge functionality, had gone live.
@ethereum The safest way to avoid this is to simply set the TTD to a very high value, and then lower it once Bellatrix is live on the CL.
@ethereum For PoW testnets, this seems to be the only way to ensure spikes in hash rate don't affect the rollout.
@ethereum For mainnet, given the large amount of hash rate, we _could_ more confidently set a TTD before Bellatrix goes live and have high assurances that it won't be hit before.
@ethereum That said, on the call we agreed that it would be bad to have a rollout process for mainnet which would be different from testnets: users and infra providers will use testnets as a way to configure things, and we don't want to tell them to do something different for mainnet.
@ethereum We then discussed whether this would cause delays in The Merge and concluded that it wouldn't. For mainnet, we can have Bellatrix activate several weeks before we are ready to set a TTD.
@ethereum The requirement for Bellatrix to go live is that the code from CL clients for The Merge should be final (at the very least for consensus changes). This is something we'd expect on the last testnet releases, so we could have those releases also include Bellatrix for mainnet.
@ethereum We would also need EL clients to be ready for mainnet at that point which, again, we'd expect for the last testnet deployments.
@ethereum To summarize, for The Merge to happen we need: 1. A beacon chain to be live 2. The BC to activate the Bellatrix upgrade, meaning it starts listening for TTD on the PoW chain 3. TTD to be hit on the PoW chain
@ethereum Our initial plan was to have a single release containing both the Bellatrix activation epoch and TTD, but now two releases will happen: one with Bellatrix and one which sets the correct TTD.
@ethereum For Ropsten specifically, this approach failed because the difficulty of the chain is very low and for a few hundred dollars per day, someone can increase it 10x over night.
@ethereum For mainnet (and any other PoW network with value), we assume that 51% of the hash rate cannot collude, so it's incredibly unlikely it would go up close to that much in a short period.
@ethereum That said, on mainenet we could face the opposite problem: we set a TTD, hash rate goes _down_, and we need to set a TTD that is lower because it is taking too long to hit.
@ethereum This is exactly what we will now be doing by default on testnets, so the community will get to "practice those muscles" early and often :-)
@ethereum For Ropsten specifically, we've now set an artificially high TTD of 100000000000000000000000, which is twice the total difficulty of the current Ethereum mainnet
@ethereum Client teams are now in the process of making releases with this value, and an announcement is expected early next week.
@ethereum π¨ If you had already updated your node to a release with the old Ropsten TTD (43531756765713534), you will need to update again, or manually override the TTD π¨
@ethereum@vdWijden The timeline for the Ropsten upgrade is: 1. Fri-Mon: client releases out 2. Mon: Beacon Chain genesis 3. Mon-Tue: blog announcement 4. June 2: Bellatrix upgrade 5. June 3: TTD chosen 6. June ~8: TTD hit on Ropsten PoW chain
@ethereum@vdWijden For other networks, the delay between (5) and (6) will be larger. We went back and forth about this on the call, but given that Ropsten is scheduled for deprecation, decided it was fine to move a bit quicker for this one.
@ethereum@vdWijden There were a few questions resulting from all of this. First was one by Potuz about the status of the JWT tokens. Quick background: the EL and CL communicate over a new API, the Engine API, which requires authentication for security, using a JWT secret.
@ethereum@vdWijden All EL teams have this implemented and most of them have it as a default, with others planning to default to it soon. This is a hard requirement for the next testnets.
@ethereum@vdWijden Next question was whether we should launch the Sepolia beacon chain now, to avoid hitting a similar situation there. General agreement was that we should, and run it through Altair ASAP, but wait before scheduling Bellatrix on it to make sure we mirror the mainnet deployment.
@ethereum@vdWijden That said, because some CL clients weren't on the call (CL call is next Thursday), we agreed to confirm that decision there. There is no rush to launch it in the next 1-2 weeks, we just want it to be live well before we fork Sepolia.
@ethereum@vdWijden There were also some questions by EL teams about what _exactly_ happens during the Bellatrix upgrade on the CL side. In short, the block format is changed and CL clients start pinging EL nodes for the TTD.
@ethereum@vdWijden The fact that the block format changes means Bellatrix does need to be introduced as a hard fork where all CL nodes upgrade at the same time.
@ethereum@vdWijden As a last point around testnets, Afri mentioned that some Goerli signers are looking to become Prater validators for The Merge on that network, and that TTD will be easy to set on Goerli given it uses Clique and not PoW.
@ethereum@vdWijden This method would allow the CL to receive a range of blocks from the ELs and optimize how it stores them to reduce storage size. It's a "nice to have" given that CL clients can mimic this today by making several calls for individual bodies and stringing the responses.
@ethereum@vdWijden This does cause more strain on the EL nodes, but given how late we are in the development process, EL teams didn't want to implement a new feature which would then need to be extensively tested. It's likely to be introduced post-merge.
@ethereum@vdWijden This raised the question around how we should version the Engine API going forward, as we'll likely introduce and deprecate methods over the years. No firm suggestions about this, but getPayloadBodiesByRange is a good opportunity for us to think through this more.
@ethereum@vdWijden Next up, we discussed the difficulty bomb π£ There was a lot of back and forth about this, so I recommend watching the livestream for the full context!
@ethereum@vdWijden@tjayrush@TokeyG We've already started to see block times slow a bit (~13.7s this week on average when I last checked). @tkstanczak then explained that he thinks we should consider a delay. He shared his rationale on Twitter yesterday too:
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak There were some client teams in support of this, and a general sense that while the merge is _almost_ done, we're not quite there yet and the bomb does add pressure.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak That said, some people were more optimistic about the rate of progress than others, and it not everyone believes we should delay the bomb.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak People also disagreed about what "unacceptable" block times are, some people thinking we are nearly there already, and others thinking 20+ sec blocks would be fine if it means keeping our focus on the merge exclusively.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak Another potential issue is that if blocks are slowed down, and then the gas limit is raised to make up the lost throughput, we could see the merge happen with larger blocks than we've currently got on mainnet.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak One idea that was floated was to combine a bomb delay with the client releases for the Bellatrix hard fork on the Beacon Chain. At that point, we will also need EL releases, so you could have them fork too, and push back the bomb.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak Realistically, this would happen at least 1 month from now, so we'll still see block times climb until then, but it would make it easier from a coordination PoV.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak In terms of the delay itself, everyone seemed to agree that 2-4 months is what would be reasonable. Doing less than that is useless, and no one felt like we would need more than that. There seemed to be a slight preference for the shorter end of that range, too.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak Again, the livestream had much more nuanced discussion, this is just the highlights π We didn't make a decision on this call, but the next step would be for someone to make a specific proposal in the form of an EIP. I suspect we'll be discussing this in the next call too.
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak By the next ACD, we'll also have had the fork on Ropsten happen, which will give us another data point as to how things are progressing!
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak First, @asn_d6 gave an update on EIP-4844, a.k.a. protodanksharding. In short, a concern with the EIP is that verifying the validity of blobs in the mempool could open up a DoS vector, as is the case with all good EIPs!
@ethereum@vdWijden@tjayrush@TokeyG@tkstanczak@asn_d6@adietrichs@dankrad We didn't discuss whether it could be included in Shanghai, as we'd previous agreed to pause such discussions until we're basically done with The Merge work and can take a more holistic view of the various proposals.
We'll discuss this more on ACD tomorrow, but TL;DR it's not an issue which could affect mainnet, and has to do with PoW testnets having very low hashrates. If you've already downloaded a Ropsten Merge client release, you will unfortunately need to upgrade again.
Worth adding that this is "malicious" in that the Ropsten miner isn't just adding hashrate (which could happen by accident/someone not knowing), but also driving down the gas limit as low as possible. Again, this isn't something we'd see on mainnet, but worth noting.
Thanks for the "support" I guess, malicious miner 0xd197?
@Butta_eth thinks they are also using McDonalds' wifi as a VPN π
@ethereum First on the call, @parithosh_j gave a recap of the two shadow forks we did last week: one on Goerli and one on Mainnet. The Goerli one happened on Tuesday and some client teams had issues, that said, most were fixed by Saturday's mainnet shadow fork πͺπ»
@ethereum@parithosh_j The mainnet work went relatively well, where we didn't see major issues and the network promptly finalized post-merge π Pari had a tweetstorm about it last week:
@ethereum First, quick shout out to @christine_dkim who has been writing recaps as well over the past few calls. I encourage folks to read her thread too for an additional perspective π
@ethereum@christine_dkim As for the call itself, the first thing we covered was the set of recent shadow forks of Goerli. TL;DR: shadow forks allow us to run The Merge on a minority of nodes on the network, but still receive transactions going to the main chain.
First up on the call, @parithosh_j gave an overview of the Kiln launch. The PoW chain went live last Wednesday, and the Beacon Chain on Friday.
@parithosh_j We were hoping to merge mid-week this week, but there was an unexpected increase in hashrate in the network. This forced us to use the override feature we built into client to set a new terminal total difficulty value. This worked perfectly! All clients merged at the new TTD!
@ethereum First on the call, we discussed the latest updates to Kiln: our upcoming merge testnet. @vdWijden has been trying to get all client combinations working and to put together a doc with instructions about how to pair them π
@ethereum@vdWijden One thing that wasn't fully aligned yet was client authentication between the EL + CL nodes. We had rough agreement on the call about how to proceed, and will make sure all clients behave similarly. In short, all calls on the auth'd port will require a token.