Thread by @mipsytipsy

This is 100% true. What always jumps out at me is that there's always "the tracing expert(s)" that everybody goes to for help on the rare occasions when they need a trace. Fluency rarely transcends the few to reach the many.

I have a different diagnosis, tho.

https://twitter.com/copyconstruct/status/1434691363451244544

I don't think it's this so much as it is that ... tracing is *inherently* a niche use case. A tracing-first approach to observability turns the world on its head for no good reason.

What most people want is the ability to slice and dice their requests,

https://twitter.com/copyconstruct/status/1434691364810158084?s=20

and occasionally trace one.

A lot of the munging engineers get so good at doing with logs is to make up for the fact that they didn't capture them as events in the first place.

The fact that they inevitably end up implementing some form of rudimentary

https://twitter.com/mhat/status/1434901549310898176?s=20

tracing with their logs shows that there *is* value in being able to visualize your events over time.

But when you ask people to hop from one tool to the next, you're going to lose 90% of the people. They'd SO much rather find a hacky way to do it all in the same tool.

Even the best tracing tools aren't exactly known for their friendliness or ease of use. People know how to get what they want out of logs, and the cognitive lift is so much less than to shift from one tool to another, harder tool.

So they keep solving for tracing in a locally-optimized way, over and over and over. Unless you're running like 200+ microservices, the costs probably won't outweigh the gains.

Don't forget that people are already jumping once, from metrics to logs.

You see a spike in your metrics. You jump to your logs to try and visually correlate by time to see what the errors meant (since you can't slice and dice or dive deeper into a metrics-based tool, having discarded all that connective tissue at write time).

And then you find a log line that looks suspicious. Then you have to copy the id over into your tracing tool, and hope it got sampled correctly so it appears there too?

That's a heavy lift. And most people don't need to do it many times a day, it's more like monthly.

That's just not often enough to give someone the chance to become fluent in it. So nobody knows how to use it well except the people who rolled it out.

When you're in the middle an intense debugging session, stopping to wrestle with a new tool SUUUUUCKS. it destroys your flow.

Any time you're asking people to learn a new tool or change the way they're doing something, as a rule of thumb, the new thing needs to be an order of magnitude better or more valuable than what they currently have.

For most teams, standalone tracing doesn't meet this bar.

People have been doing the same janky jump from metrics to logs for their entire career, btw, which is why they rarely notice how shitty and unscientific it is.

It's pattern matching and intuitive leaps, not evidence.

The right solution is to start with arbitrarily-wide structured data blobs, capture hundreds of dimensions per request per service, & append request ids, trace ids, and span ids.

Derive your metrics and SLOs from those raw events. Practice read-time aggregation, not write-time.

Then you can start at the top ("is there a problem?"), follow the trail of breadcrumbs to the answer ("the problem is requests doing xyz"), and click on any particular request to visualize it as a trace.

No jumping around, no guessing, just debugging.

Relevant:

Also, the rest of this thread + its replies were super interesting and fun to read. Always fun when @copyconstruct throws a truth bomb and walks away. 💣🚶💅