Thread by @mipsytipsy

here's another way to think about it:

monitoring is for understanding infrastructure; instrumentation + observability are for understanding your core business value.

it's how you make your code easy to understand and navigate, long after it's shipped and running in production.

monitoring is about "what's happening" from the perspective of your infrastructure components.

observability is about "what's happening" from the perspective of your users. that's why everything is gathered up around the perspective of the request as it executes.

there's a bright line here. software engineers writing features for users shouldn't have to care about the state of the infrastructure components. right? they should have to care about whether their code is doing what it's supposed to do, and about the user's good experience.

conversely, infrastructure engineers aren't responsible for transient application errors. it's their job to make sure the infrastructure service is healthy and serving traffic; all kinds of errors take place above the dotted line that they have no control over/don't care about.

generations of engineers have learned to paw through the tea leaves of low level system statistics and /proc counters, divining clues that may be related to their application errors.

this is why everyone thinks they need so many infra metrics. it's all they've ever known. <3