"observability is designing code so anyone can maintain it" is another way of saying "observability is instrumenting your code at the right level of abstraction/capturing enough detail so that you can debug by asking any new question, at any time, without shipping any new code."
good code is well instrumented code. good code makes services you can understand from the outside, without having to leap to heroics like strace or gdb or re-instrumenting it just to understand what it does.
good engineers write code observability-first. they write instrumentation first in order to understand the impact of the changes they're considering making. then they write (and check!) instrumentation along the way to make sure they shipped what they thought they did.
the best software engineers i've ever worked with, the ones i'd trust to ship on a friday night, all had muscle memory for flipping over to watch their build go live.
"did i just write what i think i just wrote?"
"did i just ship what i think i shipped?"
"who's using it? how?"
"did i just write what i think i just wrote?"
"did i just ship what i think i shipped?"
"who's using it? how?"
i fervently hope that in a few years we'll look back on the bad old days, when we used to ship code and **wait for our pagers to go off**. 😱
there are so many subtle problems you will *only* catch by looking. by forming a mental model of what should happen, and checking it.
there are so many subtle problems you will *only* catch by looking. by forming a mental model of what should happen, and checking it.
the best software engineers i've ever worked with, have a compulsive habit of poking through metrics right after they ship something.
it's freshest in your mind then, you know what you're looking for, and you know it's gonna be painful as hell if you have to find it later on.
it's freshest in your mind then, you know what you're looking for, and you know it's gonna be painful as hell if you have to find it later on.
that curiosity leads to finding shit early (*before* your users) which leads to a dopamine hit like whoaaaa.
^^ that's a senior engineer right there. the junior engineers are the ones avoiding the graphs and not looking because it's scary and they think they don't want to know.
^^ that's a senior engineer right there. the junior engineers are the ones avoiding the graphs and not looking because it's scary and they think they don't want to know.
observability is to production as tests are to pre-production.
observability means testing in production.
we all do it. you *have* to do it. you'll do it better if you face it straight. real data, real users, real concurrency, real services.
there is no substitute.
observability means testing in production.
we all do it. you *have* to do it. you'll do it better if you face it straight. real data, real users, real concurrency, real services.
there is no substitute.
