Thread by @mipsytipsy

Observability-Driven Development: write the instrumentation you need to validate that your change is working before you write the change itself.

Observability-Driven Development: where possible, ship some instrumentation to validate your hypothesis that the change is even worth making before you decide to make the change. ("how much space will this save? how expensive is this bug? how many are impacted by it?")

Observability-Driven Development: you go look at your fucking changes after you make them. did what i expected to happen, actually happen? does anything else look weird?

you need to shift from a mindset of "graphs are a thing i go look at when i get paged and things are broken", to embedding observability into your full lifecycle of deciding what to build and building it.

you should be up to your elbows in graphs every day

observability is about gaining confidence in what to build, how to build it, and that you built it. (and that people are using it, how they are using it, etc.)

it's like you're running experiments in a lab but only checking the results when an alarm sounds

if you waited for the fire alarm to go off, and then you check the whole backlog of this week's test results all at once? ... it's going to be super hard to figure out anything meaningful.

raise your hand if you've ever gotten paged and gone to investigate, and accidentally found several Serious Unrelated Problems the minute you popped the hood.

....

looking is powerful. finding problems isn't hard, it's really easy. it's SO easy because most people's prod systems are a festering mass of unexplained weird emergent behavior that everyone fears to touch -- because while finding problems is easy, finding answers has been hard.

the only answer i know, and you've seen me say this a million times: is a high cardinality, event-oriented, context-dense, tracing-aware explorable approach to debugging complex systems. like honeycomb.

otherwise it's pretty fucking hard.

i say that observability is the missing link, because lack of observability has kept us from closing the loop on so many things for so long.

chaos engineering.
testing in production.
lineage-driven fault injection.
putting software engineers on call.
... and so much more.

as long as we have to keep appealing to priesthoods to interpret dashboards, general software engineers will not have the tools they need to properly develop software, ship software, and debug software in production.

Observability-Driven Development is a liberating practice.