Thread by @mipsytipsy

Perceptive comment. Dashboards are never gonna get you to anything better than monitoring. Observability is always going to involve querying: active, novel engagement with your data.

Goes back to the part about known-unknowns vs unknown-unknowns. A refresher:

https://twitter.com/cachavezley/status/1131836041793593344

Monitoring is great for known-unknowns. Once you know a possible failure, huzzah! now you can monitor for it or write tests for it.

Observability is about having the ability to ask any question, explore or debug ANY system state, without shipping new code. Unknown-unknowns.

One of the subtle implications of this is that monitoring is a relatively passive activity. You're just pattern matching your state against a dictionary of known states. It isn't usually very taxing.

Whereas observability tooling opens the door for you to 🌸actively understand🌸 your complex systems (and their emergent behaviors!) via the process of iterative, critical interrogation.

You're actively looking for ... well you don't know what, exactly. 🌝

You formulate a hypothesis, ask a question, examine the results, ask another based on what it says. Continuously reduce the search space, follow the trail of breadcrumbs until you find the answer, the outliers, or the commonalities. Etc.

This is why people often surmise that observability is "harder" than monitoring.

It is not. (Good Lord, if you've ever tried to do exploratory interrogation using monitoring tools... like surgery with a butter knife) But it is *active*, not passive, and that can feel harder.

Most haven't yet realized that they are way past the frayed edge of the old ways. They still think better pattern matching and "single pane of glass" assurances are gonna help them understand their systems.

Or they just filling in the moat with the corpses of burnt out opsfolk.