Thread by @AlchemyPlatform

🛠 Why Alchemy stayed up today + sharing our internal playbook on managing complex node infrastructure for the community. Thread 👇

1/ As many of you already know, many Ethereum node operators and applications had issues today due to running older versions of Geth nodes.

2/ What happened? A consensus issue surfaced last night in older Geth clients. Recent releases fixed this bug, but if you were running your own nodes and don't upgrade regularly, then last night your nodes ended up on an incorrect fork.

3 / Alchemy's Developer Platform stayed 100% operational during this time due to extensive processes around testing and rolling out node upgrades. We're sharing our internal process with the Ethereum community to help everyone build more robust systems 💪

4/ Why should you care? These processes have been battle tested from over 3 years of powering billions in transactions for millions of end users in 99% of countries globally 🌍

5/ First thing to note, though Supernode (alchemyapi.io/supernode) is a novel, distributed, scalable node infrastructure, Geth and Parity (OpenEthereum) nodes are key components of Supernode so maintaining high uptime with them is absolutely crucial.

6/ Philosophy: We strive to strike a balance between reliability (older, stable versions) and introducing new features with new releases. We closely examine and test every release and update once we believe it's stable, biasing towards major releases and important new features.

7/ Step 0: We have an internal notification system that watches the Geth and Parity Githubs and automatically pings our slack when new updates are released.

8/ Step 1: Our core infrastructure engineers read through changelogs to see if there are any critical fixes that need immediate deployment.

9/ Step 2: Spin up a handful of nodes on the latest version.

10/ Step 3: Replay sample production traffic on upgraded nodes to look for breaking changes.

11/ Step 4: Add specific integration tests for all cases called out in the change log notes.

12/ Step 5: Alert customers of potential breaking changes, notifying them about any differing responses found in our tests.

13/ Step 6: If there is a major breaking change, we get feedback from customers. If this is going to be a problem, we delay or abort the update.

14/ Step 7: For major upgrades, we will run two versions of nodes in parallel and allow users to test their code on both.

15/ Step 8: Once the version is green-lighted, we spin up a whole fleet of nodes - equivalent to the entire capacity we currently have on the old version.

16/ Step 9: Blue-green deploy: run old and new nodes in parallel. Slowly shift traffic over to the new nodes.

17/ Step 10: Eng team on call and watching the deploy, ready to instantly roll back if we notice any issues.

18/ Step 11: Eng team in realtime support chats answering any questions for customers.

19/ Step 12: Eng team on call and monitoring key metrics dashboard closely over the next 48 hours.

20/ Step 13: We maintain old versions of nodes as a multi-tiered backup layer in case any unforeseen issues pop up.

21/ Step 14: Supernode has integrated Geth and Parity so worst case, even if all of one type of node goes down, we have the other to fall back on.

22/ This is a high level spec of the process. Look out for a Medium article for a more in depth explanation of our upgrade processes.

23/ Managing node upgrades is only one part of maintaining a stable, scalable developer platform. Monitoring, load balancing, and guaranteeing consistency are essential.

24/ Running your own node can be a great thing. However, it takes a lot of time and effort to manage the complexities of blockchain structure as seen here, in addition to maintaining scalability, reliability, and consistency.

25/ If you want to maintain your own nodes, reach out and we're happy to help and share in-depth lessons on node management.

26/ Many teams choose to focus on building their product instead, and that's where we can help. The Alchemy Developer Platform offloads all the complexity of node infrastructure and provides a suite of developer tools so that you can focus on what matters, building your product.

27/ The key to blockchain's success is empowering developers to build awesome, useful products.

28/ Our mission at Alchemy is to democratize access to blockchain development and enable every developer to experience the magic of blockchain!

29/ Get free access to the Alchemy Developer Platform here: alchemyapi.io/signup/

30/ Lastly, welcome to all of our new Alchemists! 🎉 Make sure you join our community discord and check out our docs here. Feel free to reach out if you need help integrating or have any questions!