, 10 tweets, 2 min read

The #1 sign that someone does not have observability is this: they lean on alarms for ⭐️EVERYTHING.⭐️

They cannot debug or introspect their systems any other way, so they abuse the FUCK out of monitoring checks and alarms.
I *just* wrote a post about this: charity.wtf/2019/09/20/lov…

Sorry if that was a little harsh. Flashbacks. Shudder.
Listen, I am not shaming anyone for doing the best they can do with the tools they have had.

It is NOT EASY trying to cobble together an understanding of a complex system using checks, alarms, dashboards, and your intuition. Sweetheart, I *know*. You do your best
But this is exactly why I feel so strongly about it. You will shred even the strongest of teams this way; it is just a matter of time. It is genuinely not compatible with human flourishing.
It worked okay for a while, in the days of the monolith and the LAMP stack. Not great, but "okay".

But your systems are getting exponentially more complex. Every dimension is high cardinality with a growing set -- services, apis, instances, containers, endpoints, storage...
Which means that the number of things that can go wrong, or combinations of things that can go wrong, or spaces between combination of things...

It's all growing a lot faster than *that* exponential curve. You cannot subject a team to this barrage of alerts and thresholds.
The ONLY way I know of to bend the cost curve down is with SLOs, observability tooling that lets you explore open endedly in real time, and limiting alerts to mirror customer impacting pain. Mostly e2e.
Page less. Not more. Page way, way, way less. WAY less.
A lightweight modern version of this worth mentioning though -- devs using honeycomb often set triggers on queries for the code they are writing, so they get a slack message if anything odd happens.

Or if like they are on call and trying to repro something for a particular user.
It is useful af, I am not denying that. It just doesn't *work* when you're doing this at scale. And it fails in the most costly and painful way conceivable: in human lives. 😕
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Charity Majors

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!