Thread by @GergelyOrosz

Today’s episode of of practical software engineering tips & tricks: load testing 3rd party services/APIs in prod.

Aka how we made sure Uber’s payments providers would also be ready for the New Years Eve surge in the early years. A thread.

Up to around 2018, Uber saw a 5-10x traffic spike on NYE. Funny enough the spike would become the new “baseline” traffic for the next September.

But the first few years this spoke would nearly cripple the company. So we started preparing ahead of time and load testing the fall.

After we’d load test most internal services and ensure they’d cope, the next failure vector: 3rd party services.

We knew what would happen on NYE and were on standby. They had no clue and though assured over email they can take a 10x load... let’s say it wasn’t always the case.

So we’d bring NYE a few months earlier. But for payments calls, we couldn’t do it with mock data. Every request meant moving real money.

So we stopped moving money for a few hours. Accumulate the backlog.

Then send it all at once, at 20x our normal rate for an extended time.

Obviously we’d do this when we had the PSP on the ready.

Almost always something wild break down that the PSP did not expect.

And the PSP wild spend the next months actually preparing for *our* surge. We just proved they were not ready, despite what they assured us. 🙃

Uber was unique in the highest business loads (and most new users) coming in at holidays, and a lot of things evolved around this characteristic.

When you have lemons, make some good lemonade. Batching prod payments requests well ahead of peak loads was a pretty good one.