Thread by @GergelyOrosz

One of the challenges at Uber was building monitoring and alerting that worked reliably.

The problem was how Uber was (is) city-based and global alerting would not catch regional (city/country-level issues).

Two stories on why this is difficult:

1. A PayPal employee on a Japan business trip alerted us in 2016 that PayPal is not working there. He was right: it wasn’t working for 2+ months, across the country. How did we miss it?

There were 20 PayPal trip attempts/day only, and Japan was one of 60+ countries.

In the global scale, this accounted for a tiny fraction.

So we did what makes sense: added country-level alerting.

First, this became a data cardinality problem. 1,000 cities x 15 payment methods... not trivial to track all. We settled on countries & top cities.

With country/city alerting, now the sparseness of the data became a problem. It’s still to alert reliably on 20 data points/day.

The new system worked, but it was really noisy for months, until we “tamed” it.

2. On 8 Nov 2016, a team member in Amsterdam got paged to an outage in India. Cash trips were not working. They debugged, but all was fine... except for the data. Almost NO cash trips.

Turns out, this was the day of cash demonetisation. This would not be the first such alert:

Every year we’d have many alerts from countries and cities tied to real-world events. Hurricanes, large concerts, local marketing events. All of them alerted the system as unusual, and paged engineers. But they were “normal” in the local context.

Monitoring & alerting is an area you can go down the rabbit hole, especially in a multi-country, multi-city, real-world-bound context.