Something just clicked for me this week: how much the transition from monitoring to observability is about a shift in perspective, from third-person observer to first-person narrator.
And how much this says about why putting software engineers on call is hard.
And how much this says about why putting software engineers on call is hard.
Monitoring has traditionally been one piece of software keeping an eye on another piece of software, and describing its state. This goes for white box monitoring too, not just black box.
Whereas observability requires a different level of abstraction, one that practically demands it be an instrumentation game. This is a firsthand perspective -- the code explaining itself back to you, your future self, your future team.
Obviously it is going to be event based, because that's how requests traverse code paths -- certainly not in disconnected metrics and counters.
But what does this have to do with software engineers on call?
Well, we have 20+ years of building monitoring tools for ops eyes. Ops thinks in terms of system resources and /proc values. SWEs think in terms of lines of code and code paths.
Well, we have 20+ years of building monitoring tools for ops eyes. Ops thinks in terms of system resources and /proc values. SWEs think in terms of lines of code and code paths.
Asking software engineers to look at an old school dashboard of system resources and draw any kind of coherent line back to the problem they are trying to investigate is... unreasonable.
The amount of knowledge, intuition and scar tissue required to interpret those is too large.
The amount of knowledge, intuition and scar tissue required to interpret those is too large.
In my experience, asking software engineers to be on call when they have a debugging tool like honeycomb -- one that speaks in variable names and request path and context broken down by event -- is supremely reasonable.
When they don't, then you start hearing the gripes about how you're asking them to do two jobs, or how helpless they feel when the pager goes off.
It's not two jobs, but you *are* asking them to store all the debugging wisdom and conventions of two roles in their brain. Hard.
It's not two jobs, but you *are* asking them to store all the debugging wisdom and conventions of two roles in their brain. Hard.
The solution isn't to stop letting engineers support their own code. You *need* those tight feedback loops to be a high performing team.
The solution is to empower them with tools to make their jobs possible for mere mortals. First-person, event-driven, instrumentation based.
The solution is to empower them with tools to make their jobs possible for mere mortals. First-person, event-driven, instrumentation based.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
