Some commentary on the #Ropsten testnet upgrade to Constantinople: what happened this past weekend, where we’re at now, and what this means for Constantinople. Hopefully this helps get everyone on the same page. It’s long but there’s a lot of information to convey!
The Ethereum core devs have been working on the Constantinople hard fork for months. The two major clients, parity and geth, were feature complete nearly a month ago. See the progress tracker here: github.com/ethereum/pm/is…
On the core devs call on Sept. 28, the decision was made to hard fork Ropsten at block 4.2M on or about Oct. 9. There were some concerns that we were rushing the hard fork, but two factors drove this decision. github.com/ethereum/pm/bl…
1. @EFDevcon is upcoming and we wanted the fork to happen before everyone is traveling.
2. We’re not that concerned with borking the testnet anyway. That's what a testnet is for. In fact, we thought that it might be good to test the impact of a long fork and chain reorg on a network with live users and dapps.
Brief aside on Ropsten: It’s a PoW testnet that includes both geth and parity nodes. PoW testnets will always have issues since mining is resource intensive but the testnet ether that miners receive is valueless. It’s been attacked before: github.com/ethereum/ropst…
A PoA testnet like Kovan or Rinkeby makes a lot more sense but for now parity and geth do not speak the same PoA protocol so PoW is the only way to have them both on the same testnet. Fixing this is a work in progress: dev.to/5chdn/the-grli…
Back to the hard fork. A last-minute issue was discovered in the changes to CREATE2 and the decision was made to push the hard fork back by ~5 days to block 4.23M (~Oct. 14) to give the core devs time to make updates. github.com/ethereum/go-et…
The clients were fixed, a bunch of upgraded nodes were spun up, and everyone hunkered down to wait for the fork. The countdown started on Saturday afternoon UTC time. gitter.im/ethereum/AllCo…
The hard fork kicked in around 15:21 UTC time on Saturday, or ~1 day earlier than expected since hash power and thus block times on Ropsten are notoriously hard to predict. However, after this, the fork froze and no new blocks were mined.
Either there were no miners running upgraded software or else the difficulty was too high and there weren’t enough. (I’m not entirely clear on what happened.) This was probably our fault as core devs. We should’ve been prepared with some upgraded miners ready to mine the new fork
@infura_io, github.com/atlanticcrypto and others heroically fired up some Ropsten miners but the chain is long so they took a while to sync and start mining. Blocks began getting mined ~2 hrs later.
One thing I want to make abundantly clear: the lack of miners on Ropsten had *nothing* to do with the Constantinople issuance reduction. As described above, testnet mining is purely altruistic and most major mining pools do NOT mine on Ropsten anyway.
At block 4230605 something more serious happened: parity and geth disagreed about the gas that should be charged for a tx. Here is the offending tx: ropsten.etherscan.io/tx/0x2aa56c25d… gitter.im/ethereum/AllCo…
(For a laugh, view the transaction data as UTF-8 to see what broke the testnet!)
(For a laugh, view the transaction data as UTF-8 to see what broke the testnet!)
This is a consensus bug and it caused geth and parity to fork. (So, if you’re keeping count, there were now three Ropsten chains: Ropsten Classic, Ropsten geth, and Ropsten parity 🤓)
@realLedgerwatch , @sorpaas, @mhswende , @5chdn and others worked tirelessly through the weekend to identify and fix the issue. @mhswende discovered the issue in parity here yesterday afternoon: gitter.im/ethereum/AllCo…
The bug involved the calculation of the gas refund for SSTORE in parity and was a simple integer overflow. @sorpaas quickly deployed a fix. Here’s the fix: github.com/paritytech/par…
Work is ongoing to test and roll out the fix. Lots of forensic work remains to be done to understand not only how the bug happened but how it wasn’t caught by tests, how the EIP process might have contributed to the issue, how well we communicated and responded to the issue, etc.
This begs the important question: what happens to Constantinople now? As @5chdn and @jutta_steiner have pointed out, it’s likely going to be delayed since we need to be as conservative as possible about mainnet upgrades. But there was no set date for it, anyway.
We’ll likely want to run another testnet upgrade and see it running successfully on the testnet for weeks before considering a mainnet upgrade. Unclear yet whether this will happen on Ropsten or on a new testnet, and whether or not we want to rewind Ropsten.
We’ve learned a great deal throughout this process and will continue to for some time. For me, it’s been exciting and educational to see this process play out and to see the community come together and respond to the issue, even on a testnet and over the weekend.
To reiterate: this is precisely what a testnet is designed for, esp. an active one like Ropsten, and this is precisely why upgrades happen on the testnet first. Seeing this process unfold makes me more confident than ever that mainnet upgrades are handled responsibly.
One final thought: I’ve only been an observer throughout this process and this is my attempt to summarize what’s happened. Please correct omissions or errors and, again, massive kudos to those who did the actual work! @5chdn, @mhswende, @sorpaas, @realLedgerwatch and many others.
