Thread by @mipsytipsy

went rummaging through slack history, realized it's been three years since i first googled the definition of 'observability' and found to my shock that:

a) it *had* a technical definition
b) it was everything i had been struggling and flailing to describe for the past six months

so, i wrote up a three year retrospective of how the term has been used and misused, claimed by vendors both good and evil, as well as the technical requirements for systems observability ... and why high cardinality is at the root of everything. thenewstack.io/observability-…

warning, it's long; and possibly of interest to only like me and five other people. but the important thing to note here is this.

the reason we got attention, in a NOTORIOUSLY packed space, is because we reflected the real pain and anger that engineers like us are feeling.

so many engineers have been shelling out ungodly sums of money to newrelic and datadog etc, despite the fact that those tools increasingly do not solve their problems, and solve a little less and less every month.

we live in a high cardinality world.

we live in a world where canned answers are increasingly ineffective.

we live in a world where you must instrument with intent, not rely on magic.

we live in a world where user experience is the only metric that matters.

we live in a world of cheap hardware and expensive time.

the metrics, logs and APM providers got their start back when they did a pretty good job of solving your problems. they've been printing money left and right ever since.

they weren't designed for a distributed world, but they've been making too much money to care.

but the undercurrent of dissatisfied folks who knew they were wasting their cash, knew their problems weren't being solved, were dubious of "high cardinality is impossible" -- has grown rapidly.

you people are why we are still around today, and gaining steam fast. 💙

turns out people don't particularly care for being gaslit by their software vendors. nor told that what they want they can't have, but how would they like thousands of dashboards instead??

vendors keep papering this gap by releasing features named e.g. "Infinite Cardinality"

instead of releasing, say, actual support for high cardinality. (you are not fooling anyone)

this smug stasis seems to be what happens when a category is raking in so much money, they don't notice or care when the world has moved on.

in conclusion, thanks to everybody who has given us a try, given us your feedback.

i too aspire to print so much money i don't have to care about anyone's feelings, but until then... we'll be over here trying to build the shit you need to understand your systems. 🧸🐝