OH GOD DO NOT DO THIS THING
The #1 sign that someone does not have observability is this: they lean on alarms for ⭐️EVERYTHING.⭐️
They cannot debug or introspect their systems any other way, so they abuse the FUCK out of monitoring checks and alarms.
The #1 sign that someone does not have observability is this: they lean on alarms for ⭐️EVERYTHING.⭐️
They cannot debug or introspect their systems any other way, so they abuse the FUCK out of monitoring checks and alarms.
I *just* wrote a post about this: charity.wtf/2019/09/20/lov…
Sorry if that was a little harsh. Flashbacks. Shudder.
Sorry if that was a little harsh. Flashbacks. Shudder.
Listen, I am not shaming anyone for doing the best they can do with the tools they have had.
It is NOT EASY trying to cobble together an understanding of a complex system using checks, alarms, dashboards, and your intuition. Sweetheart, I *know*. You do your best
It is NOT EASY trying to cobble together an understanding of a complex system using checks, alarms, dashboards, and your intuition. Sweetheart, I *know*. You do your best
But this is exactly why I feel so strongly about it. You will shred even the strongest of teams this way; it is just a matter of time. It is genuinely not compatible with human flourishing.
It worked okay for a while, in the days of the monolith and the LAMP stack. Not great, but "okay".
But your systems are getting exponentially more complex. Every dimension is high cardinality with a growing set -- services, apis, instances, containers, endpoints, storage...
But your systems are getting exponentially more complex. Every dimension is high cardinality with a growing set -- services, apis, instances, containers, endpoints, storage...
Which means that the number of things that can go wrong, or combinations of things that can go wrong, or spaces between combination of things...
It's all growing a lot faster than *that* exponential curve. You cannot subject a team to this barrage of alerts and thresholds.
It's all growing a lot faster than *that* exponential curve. You cannot subject a team to this barrage of alerts and thresholds.
The ONLY way I know of to bend the cost curve down is with SLOs, observability tooling that lets you explore open endedly in real time, and limiting alerts to mirror customer impacting pain. Mostly e2e.
Page less. Not more. Page way, way, way less. WAY less.
A lightweight modern version of this worth mentioning though -- devs using honeycomb often set triggers on queries for the code they are writing, so they get a slack message if anything odd happens.
Or if like they are on call and trying to repro something for a particular user.
Or if like they are on call and trying to repro something for a particular user.
It is useful af, I am not denying that. It just doesn't *work* when you're doing this at scale. And it fails in the most costly and painful way conceivable: in human lives. 😕
