,
8 tweets,
2 min read
Read on Twitter
It's not often I realize I may have *under*-ranted about a particular topic, esp in the o11y space, but i just realized I've been letting pre-aggregated data off the hook in one very critical way: cost.
I have talked at length about how metrics strip context, how aggregation defines (and locks you into) only the questions you could predict having to ask, which is why metrics can't deliver observability (which encompasses unknown-unknowns).
But I haven't talked about cost. And it's ironic, because the metrics kids like to rag on the logging/events providers for being. And it's true: storing rich, wide events does look costly next to "cheap" metrics.
But you know what's *really* costly?
Having to store another copy of everything that happens for every. single. fucking. use case.
Having to shell out for yet another tool every time you want to interrogate your systems in some marginally different way, or by a different team.
Having to store another copy of everything that happens for every. single. fucking. use case.
Having to shell out for yet another tool every time you want to interrogate your systems in some marginally different way, or by a different team.
A request enters your system, pings around a bit and exits -- once.
And you store the information about it in how many different places?
* monitoring/metrics
* logs
* traces
* APM
* exception tracker
* B.I.
* front end metrics
*...
And you pay for it again, every time.
And you store the information about it in how many different places?
* monitoring/metrics
* logs
* traces
* APM
* exception tracker
* B.I.
* front end metrics
*...
And you pay for it again, every time.
I'm not one to skimp on o11y spend; I think you should spend 10-30% of your infra budget on tools to understand it and answer questions about it.
But that's 10-30% *total* spend. Most of these tools are priced to cost you 10-30% *each*.
But that's 10-30% *total* spend. Most of these tools are priced to cost you 10-30% *each*.
Those tools would be reasonably priced if they were the only tool you need.
But they aren't.
Because they pre-aggregate.
You can only achieve observability if you have access to raw events, so you can ask new questions at will without shipping new code.
But they aren't.
Because they pre-aggregate.
You can only achieve observability if you have access to raw events, so you can ask new questions at will without shipping new code.
And while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events.
The reverse is impossible.
The reverse is impossible.
