This is a great question, and I have made a note to write an article and chapter about it.
The tldr: I think of observability as being a tight feedback loop around building, fixing, changing code in systems, which relies on your having *some* ability to instrument.
The tldr: I think of observability as being a tight feedback loop around building, fixing, changing code in systems, which relies on your having *some* ability to instrument.
But maybe this a tension between the technical definition I espouse (can you ask+answer novel questions to understand any system state), and what constitutional lawyers would call its penumbra.
There's no reason you *must* be able to instrument, just in practice you usually do.
There's no reason you *must* be able to instrument, just in practice you usually do.
There are certainly ways of assembling and approximating this data without new instrumentation, *if* the software is emitting it in the first place.
I am thinking of what I've done with databases in the past, like MongoDB and Mysql.
I am thinking of what I've done with databases in the past, like MongoDB and Mysql.
For Mysql, we approximated o11y through an unholy mix of sampled full query logging, streaming over tcpdump and reconstructing transactions on a separate host, plus a cron job that connected to the terminal and dumped stats every could seconds.
Gross, but works pretty well.
Gross, but works pretty well.
