Thread by @mipsytipsy

it's a bit counterintuitive, but the better-instrumented and the more mature your systems are, the fewer problems you'll find with automated alerting and the more you'll have to find by sifting around in production by hand.

https://twitter.com/arclight/status/1454013188366798849

Becoming well versed in exploring your systems via production tooling has never been a more important part of being a good engineer.

It's also never been *easier* to derive rich insights. (why, in MY day, all we had was sar and *stat AND WE LIKED IT)

https://twitter.com/arclight/status/1454014973659729921?s=20

Apologies to whoever originally made this awesome gif about testing in production, but it holds just as true for alerting and debugging. 🙃

The overwhelming majority of the failures in your system and the bugs in your code will never create problems that rise to the level of our paging alerts.

And thank heavens for that, or it would be buzzing *nonstop*

But that doesn't mean they aren't /there/. And it doesn't mean they aren't a constant drag on your performance and productivity. These failures manifest in bug reports, customer frustration, flakiness, and mysterious behaviors that no one can reproduce.

We talk a lot about how important it is to reduce the pager volume, to stop paging on symptoms, to move to SLO-based alerting. All of this is absolutely true.

But we don't always talk about the fact that you need to *pair* that cleanup with time and tooling upstream.

Imagine that you just shipped a small feature or change to your storage engine. It passes tests, all alerts are green. Do you beam proudly and move on?

No, you *poke* at it for a while. You look at your instrumentation to see if any counters are incrementing correctly,

you watch the count(*) of new users as they're clicking on it, with full row details so you can watch for any odd patterns (exclusively from one device type?) on the success events.

you watch the count(*) and contents of any errors or unknown events. maybe you set a watch on it,

so you'll get a slack ping if they start skyrocketing for some reason. Maybe you peek at the memory and cpu usage.

You wait a couple days, and you come back and check on it again as part of your morning routine, along with checking tickets and deciding what to do today.

You keep an eye out for outliers of any type, until you're satisfied it's baked. Because you *know* that production is the only place interesting bugs show up, and you know that takes time, and you know they won't show their rotten little faces in any aggregates

Deploying software, after all, is not like flipping a switch. It is not blue or green, or off and on. It is the *beginning* of the process of validating your software in production.

I feel like the only reason this isn't as universal as running tests is that until recently, the tooling wouldn't *allow* you to ask the kinds of questions you need to ask to stick the landing.

You need raw rows, high cardinality, high dimensionality and traces to ask them.

With modern observability tooling, though, it is terribly easy and doesn't take much time at all.

Not going back to check on your code is like flying blind. It's the last and most important step in developing software. It will never again be this easy to find your bugs.

The cost of finding and fixing bugs rises *exponentially* the longer it takes between writing them and finding them.

I can't find the fb eng article I used to cite, but this one isn't bad.

deepsource.io/blog/exponenti…

So, if:

* you have to run in prod to find the bugs
* it gets exponentially more expensive to fix them

..you see how profound that moment is, when you first turn your code on in prod. Anything you do to empower engineers at that point is probably the best investment you can make