Yes!!! Monitoring tools have gotten very good at efficiently checking for a vast library of previously-seen scenarios, collecting telemetry on known-unknowns.
This matters. A great deal of engineering is moving things from the unknown-unknowns pile to the known-unknowns.
This matters. A great deal of engineering is moving things from the unknown-unknowns pile to the known-unknowns.
Engineering problems are *hard*; they take creative thinking, they are inherently open ended. These are the unknown-unknowns.
And it's our job to make them known unknowns as soon as we can, so they can be monitored, automated, etc. So they become predictable.
And it's our job to make them known unknowns as soon as we can, so they can be monitored, automated, etc. So they become predictable.
Every monitoring check or dashboard is the artifact of some past debugging session, the answer to some long ago question.
They can be useful to us, in much the same way as our test suite can -- by minding the edge cases and protecting us from regressions.
They can be useful to us, in much the same way as our test suite can -- by minding the edge cases and protecting us from regressions.
But flipping through dashboards should not be confused with debugging new questions, any more than scrolling through test output.
When you scan your dashboards, you are pattern matching current behavior against old answers, not methodically following a trail of breadcrumbs.
When you scan your dashboards, you are pattern matching current behavior against old answers, not methodically following a trail of breadcrumbs.
A generation has grown up having only monitoring tools to understand their systems. That, plus the persistent separation of those who develop the systems and those who run them, is why our systems are so poorly understood.
The devops movement and push for software ownership are salutary and necessary efforts.
But without the right tools, they will be insufficient. As @Mads_Hartmann said: if you want resilience, it demands observability. And not in the "generic synonym for telemetry" kind of way
But without the right tools, they will be insufficient. As @Mads_Hartmann said: if you want resilience, it demands observability. And not in the "generic synonym for telemetry" kind of way
