,
7 tweets,
2 min read
Read on Twitter
A: Yes, there are many aspects of instrumentation and observability that are worth investing in, even for monoliths. Distributed systems are just where the pain of *not* doing so becomes prohibitive.
This is largely for historical reasons. Due to hardware cost, we optimized for skimpy metrics data, and generations of engineers grew up fitting their thinking and tooling around scarce data types like metrics instead of richer event-trace data.
It is categorically easier to debug and understand your systems with o11y than with scattered, aggregated/averaged/aggregated metrics.
It's incredibly hard to form a mental reconstruction (or hazard a few guesses) about why events are erroring when all you have are dashboards.
It's incredibly hard to form a mental reconstruction (or hazard a few guesses) about why events are erroring when all you have are dashboards.
It's much easier to simply ask a straightforward question. Which you can do if you gathered the data by event in the first place.
Instead of seeing some spikes and running through a library of guesses as to the cause, you just ask. Compare:
Instead of seeing some spikes and running through a library of guesses as to the cause, you just ask. Compare:
Old way being: "I see a latency spike. Is it mysql timing out? Is it a memcache issue? Is it us-east-1? Is it coming from apps running an old version of the code?"
as you flip through dashboards one by one to check for data to support your hypotheses
as you flip through dashboards one by one to check for data to support your hypotheses
With events, it goes more like this: "I see a latency spike. I will circle it with the mouse, and -- aha, those errors are all timeouts to the /export endpoint from a test user exceeding the upload limit"
(ok I sorta cheated, that's using bubbleup instead of iterating)
(ok I sorta cheated, that's using bubbleup instead of iterating)
The point is: you don't have to guess or imagine sprawling abstractions. The data
Is right in front of you.
It takes a little work to unlearn old habits. But new grads and people who haven't done hard time in ops tend to find it much easier and more intuitive than the old ways.
Is right in front of you.
It takes a little work to unlearn old habits. But new grads and people who haven't done hard time in ops tend to find it much easier and more intuitive than the old ways.
