Thread by @mipsytipsy

this is a lovable mess of a wall of text about observability and complex systems that nevertheless manages to hit all the top consequences and learnings: blog.mi.hdm-stuttgart.de/index.php/2019…

in fact i love it enough that i'm going to strenuously argue with it.

(while reminding myself that it's a measure of success for people to take the things you arrived at through blood sweat and tears and consider them "things everybody knows and says")

but: there are not "three pillars" of observabiity. there is only one data structure that underpins observability: the arbitrarily-wide structured data blob. consider:

* metrics are trivially derived from it
* logs are a sloppy version of it
* traces are a visualization of it

you can *approximate* a sloppy-ass version of observability using metrics, logs and traces. but you will spend an assload storing your data three different ways.

or you can just store it in arbitrarily-wide structured data blobs at the origin, and store it once. ob$ervability.

it's kind of fucking remarkable to me that it's taken this long for folks to wake up to the fact that they are spending many multiples of what they should be, because of the inefficiencies of their "three pillars" of data sources.

thought viruses are powerful tho i guess

why does the arbitrarily-wide structured data blob underpin observability, and not metrics, logs and traces?

* context is everything; the blob knits values together into a single event
* context lets you reason about your systems in ways that decoupled metrics never can

* you could call these blobs "logs" but "logs" are historically much messier than this. we require discipline.
* observability demands flexible read-time computation, not string searches
* structuring your data allows you to run computations across events

* your events must be "arbitrarily" wide, because you don't *know* in advance how much context you will have to attach to the event
* which rules out the metrics data type (ints with tags), or even any strict indexed schemas

bottom line: observability is about unknown-unknowns, or asking *new* questions of your data without shipping new code.

this means you must be able to perform read-time computations across raw structured events. you *cannot* get that shit from metrics and/or logs and/or traces.

so EITHER you buy the "three pillars of observability" definition,

OR you buy the "observability is about unknown-unknowns" definition, the one grounded in control theory.

they are actually quite incompatible.

you know which one i subscribe to: the technical definition that provides new value for debugging and understanding complex distributed systems.

(fuck the three pillars ✊)

i know i know this was way down in the weeds for most of you. and on a sociovendorpolitical level sure, we can smudge over the differences and play nice. guess i'm just spoiling for a fight tonight and this one was handy.

peace ✌️