Thread by @mipsytipsy

I was looking for a particular observability quote so I could cite the author, instead found a bunch of snark & shit like this, with old school monitoring dudes saying any other definitions are "just marketing" (sneer).

Tiniest smidge of a rant incoming.

https://twitter.com/rakyll/status/947953311184850945

Yes, observability has been around a while. Twitter had an observability team ten years ago, etc. I'm not sure when the "three pillars of observability" thing was coined, but it was quite a while ago. When we started honeycomb 2.5 years ago, I had never heard of it.

The hardest thing about year one of honeycomb by FAR had to do with language, as in trying to figure out how to talk about what we were doing. Every technical term around monitoring and data comes *so* grossly overloaded and saturated with generations of baggage and assumptions.

We knew what we were doing was radically different than the 20+ years of time series aggregated metrics lineage. We only knew it because we had lived through it and seen it change everything about how we worked, at Parse/Facebook.

But we couldn't figure out how to describe it.

Nothing we tried resonated with people. Every phrase we tried was used by someone else in the space. It was too generic and applied to everyone (debugging with structured data), or too specific and no one understood it (cardinality problems).

One night, very late, I noticed the term "observability" and began reading on it. Not the metrics/logs/traces definition, but the control theory definition: how well can you understand the system by asking questions outside of it?

I got *so* excited. IT MADE SO MUCH SENSE.

*So* many things fell into place once we had a technical term to describe what we were doing. How modern systems are different than past systems. Why observability is different from monitoring. Why the technical decisions we had made mainly on instinct had been so critical.

We were really, really shaky in the early days (things you're never supposed to say as CEO). But after we learned about observability our mission became more clear: building tools that let you explore a system's internal workings by exploring, asking questions from the outside.

That described perfectly our experience running large distributed systems and platforms.

You can't ssh into every host and run strace or gdb to diagnose. You have to instrument and capture the right level of abstraction so you can reason about it and drill down with the tool.

We've been excitedly learning more and more about what technical traits fulfill this definition ever since, right along with you:

* High cardinality
* Wide events, not metrics
* No pre-aggregation -- sample for cost control
* Tracing
* Rich instrumentation
* Exploratory
* etc

It's all about empowering the developer to ask any arbitrary question of their system from the outside, without having to predict it in advance, without shipping new code.

Cardinality limits make this impossible.
Metrics make this impossible.
Pre-aggregation makes this impossi

... etc.

Lots of folks have begun talking and thinking and building around observability lately through this lens. There's been so much pent-up energy and hard work around debugging complex systems, it just needed a technical term to rally around && so we could find each other.

That's when the folks who were attached to the previous definition started to take note and get pissed at me.

... omg, @rakyll just *blocked me*? for quoting her? wtf. holy shit. i thought we were friends and so was extra respectful in this thread. i'm ... sorry?!

well motherfucker.

(little bit jarred here)

All I'm trying to say is this: language evolves. Technical terms arise, mutate, and fade. O11y once meant metrics/logs/traces, but that's not as precise or useful as the control theory definition, so we should let the old one fade.

In conclusion, consider the term "metric". Once upon a time it meant something very generic ("a method of measuring something"), but we redefined it to mean something very specific: a single number (count, distribution, gauge, rate) with a type, and tags for grouping.

The more specific a term, the more it can do for us as engineers.

I salute you, old guard, for all you've done. But from those of us laboring to make complex systems tractable: please let "observability" go. Let it develop a precise technical meaning, just like "metric" did.

i guess being blocked by one person i look up to is enough for me for one day. 😔 why do i even do this? good night.

(not sure if she deleted the tweet or not, but it was something about how observability shouldn't be defined by what one company is building (HUGELY AGREE) and has always meant metrics, logs and traces and shouldn't be redefined (disagree))