Last week, two L3s (Degen, Proof of Play) were down for ~2 days, and we now have the full post-mortem report from @conduitxyz .
Here's an overview of what happened and what we can take away from this incident:
Quick TLDR: This was an edge case caused by a combination of a large increase in on-chain activity and a misconfigured attempt to push the bounds of what a DA layer could handle.
- A change was applied to reduce txn costs due to the amount of volume both chains saw
- The change increased a batch size parameter to what was assumed to be a safe value
- Unfortunately, the data availability layer could not process the larger batches, causing the posting to fail
- Testing simulations didn't catch this issue because the simulated txns were compressed more than expected
- Because posting didn't happen, a re-org occurred
- However, the re-org was so large it corrupted an internal db, which forced Conduit to resync each chain from genesis (Block 0)
How can this be prevented in the future?
Conduit has pledged to increase their alerting and monitoring to detect issues faster. Additionally, safety harnesses will be added to the code to prevent this particular from scenario from happening. They've also stated that they will work on hard on improving chain simulations (this is a really hard problem). All of these will reduce the likelihood for future issues.
Was this the nail in the coffin for L2s and L3s?
I think this was just the start for L2s/L3s. The main driver of the problem was having too many transactions, which is a good thing to deal with in a vacuum.
However, it's extremely important that the teams running blockchains prioritize uptime and liveness (both the actual chain and RPC/nodes). Conduit is one of the most technically excellent teams in the space. I wholeheartedly believe they will get better from this incident.
These chains still seem very centralized though...
L2/L3s in their current forms are still quite centralized. Most of them operate with a single sequencer and are nearly unusable when the sequencer goes down. Improvements to both the force inclusion UX and sequencer decentralization will add more resilience to rollups.
In this particular scenario, the chain was technically always live - it just wasn't possible to submit a txn through the sequencer.
Conclusion
Building blockchains is hard. We've seen L1s, L2s, and L3s all run into scaling issues this year.
And that's a good thing.
We're pushing the boundaries of what's possible and improving the tech so we can continue to move the world on-chain. Better processes, better monitoring, and more decentralization will lead to more resilient systems. Props to Conduit for being transparent and quick to act through all of this.
Here's an overview of what happened and what we can take away from this incident:
Quick TLDR: This was an edge case caused by a combination of a large increase in on-chain activity and a misconfigured attempt to push the bounds of what a DA layer could handle.
- A change was applied to reduce txn costs due to the amount of volume both chains saw
- The change increased a batch size parameter to what was assumed to be a safe value
- Unfortunately, the data availability layer could not process the larger batches, causing the posting to fail
- Testing simulations didn't catch this issue because the simulated txns were compressed more than expected
- Because posting didn't happen, a re-org occurred
- However, the re-org was so large it corrupted an internal db, which forced Conduit to resync each chain from genesis (Block 0)
How can this be prevented in the future?
Conduit has pledged to increase their alerting and monitoring to detect issues faster. Additionally, safety harnesses will be added to the code to prevent this particular from scenario from happening. They've also stated that they will work on hard on improving chain simulations (this is a really hard problem). All of these will reduce the likelihood for future issues.
Was this the nail in the coffin for L2s and L3s?
I think this was just the start for L2s/L3s. The main driver of the problem was having too many transactions, which is a good thing to deal with in a vacuum.
However, it's extremely important that the teams running blockchains prioritize uptime and liveness (both the actual chain and RPC/nodes). Conduit is one of the most technically excellent teams in the space. I wholeheartedly believe they will get better from this incident.
These chains still seem very centralized though...
L2/L3s in their current forms are still quite centralized. Most of them operate with a single sequencer and are nearly unusable when the sequencer goes down. Improvements to both the force inclusion UX and sequencer decentralization will add more resilience to rollups.
In this particular scenario, the chain was technically always live - it just wasn't possible to submit a txn through the sequencer.
Conclusion
Building blockchains is hard. We've seen L1s, L2s, and L3s all run into scaling issues this year.
And that's a good thing.
We're pushing the boundaries of what's possible and improving the tech so we can continue to move the world on-chain. Better processes, better monitoring, and more decentralization will lead to more resilient systems. Props to Conduit for being transparent and quick to act through all of this.
@conduitxyz Full incident report:
https://x.com/conduitxyz/status/1793039239128522874
• • •
Missing some Tweet in this thread? You can try to
force a refresh
