Thread by @WietseWind

I have some explaining to do.

The past couple of days were challenging when it comes to the @XummWallet / XRP Ledger infrastructure I'm responsible for.

The amount of XUMM users & clients connected to xrplcluster.com has been rising and rising. Scaling was required.

A couple of weeks back we scaled the XUMM platform, confident that we'd be able to handle the user growth for some time to come. We also made sure that future scaling would be easier.

Right after scaling, our cloud provider had multiple outages. That never happened before.

The communication of our cloud provider and the amount of sudden issues resulted in an emergency migration. This came with one hour of downtime. I personally kept you all posted on our status page: status.xrpl-labs.com.

Our status page had been a boring, green page for years.

After our cloud migration, yet again we were confident that the worst was behind us. However, as Murphy's law states: "Anything that can go wrong will go wrong."

After the XUMM platform was scaled for growth (500 » » 2000 » 5000 new users per day!) we ran into new problems.

XUMM, just like many other ecosystem participants (like @Ledger, websites, platforms, tools, exchanges) needs to connect to XRP Ledger nodes. "rippled" is the software that powers the XRP Ledger. Responsible for consensus, history, transactions: "the blockchain".

The XRP Ledger software, "rippled" has been tested with high amounts of transactions (throughput tests). During these tests it became clear that the XRP Ledger can handle a significant amount of transactions submitted by a client.

Today's production workload is different: instead of one client submitting many transactions, there are many clients submitting a small amount of transactions. The many clients do, however, read a lot of data from the "rippled" nodes: Trust Line balances, offers, order books.

To handle the amount of requests for the high amount of users (clients), more scaling was required. This time not the XUMM platform, but "xrplcluster.com": the cluster that provides access to the XRP Ledger for XUMM and many other companies, developers & tools.

Scaling blockchain nodes is harder though. A lot of testing, tuning & coding was required. Thanks to some of the well known & bright minds in our ecosystem, working behind the scenes day and night for days, the situation seems to be under control again.

The cluster was upgraded:
- From 8 full history nodes to 12 nodes
- 9 new dedicated non full history nodes were added
- 2 transaction submission nodes were added

On top of adding capacity, I wrote code to automatically orchestrate seamless client handover between nodes.

New code is monitoring the entire cluster on a per host per host type basis, and if a host is too busy, misbehaving (overloaded) or simply down, clients will immediately be routed to another node. Also: anomaly detection, transaction offloading (e.g. if full history isn't needed)

The first results are promising, and after many stressful days I am hopeful that I can finally get some good night's sleep again, knowing that the cluster will take care of itself now.

While this scaling and "patchwork" (or automation, if you like) may solve things now, (...)

(...) it will take a lot of effort, testing, profiling, searching, discussing and coding to optimize many components powering our ecosystem to keep up with the growth we're currently seeing. I'm confident that the bright minds amongst us will join forces & make this happen.

After very very stressful weeks I feel the worst is behind us. I am very sorry for the horrible user experience some of you have had the past days using XUMM or other XRPL tools. Sorry. Really. 😚

I have to remind myself (and you?) that this is all happening because of growth we could have only dreamed of just months ago.

Please take a moment while using XUMM, the XRPL, transacting, using the DEX to realize there are many geeks working day and night to make it possible.

And now: onwards 🚀

😚 @bharathchari @nbougalis @XRPL_rabbit and many others for providing node capacity, offering to help, debugging & fixing alongside of me & sending kind words my way. @XummWallet users, community: you are awesome. Thank you for keeping my spirits up & being so understanding.