Been having a lot of conversations about distributed tracing lately and whether the products that exist today (either OSS or propriety) provide enough value to justify the implementation costs. My answer: the costs are not justified and most orgs should not deploy. (Thread)
Let me be completely clear: I think the data that tracing systems provide is invaluable. The issue is that the current state of the art UIs do not expose this data in a way that is useful to the average developer.
Sure, looking at a pretty trace page is fun, but how does a user find a trace that is relevant to their debugging needs? The honest answer is that they often can't or are confused by the UI and give up.
I've been saying this for a while, but we need to completely rethink infra/obs UIs. Instead, most recent product offerings are just providing better stats, better logging, or better tracing. They are better, but are not changing how devs interact with the system as a whole.
Instead of the trace view being the first screen that a user sees, it should be the 4th. The UI needs to guide users through viewing a system topology, asking questions, and then gradually drilling down, until a trace view becomes the right screen/abstraction to interact with.
So let's come back to deploying a tracing system. My general experience is that these systems are costly to deploy both in human resources as well as HW. They are also hugely hyped as the tool that will unlock "real" distributed systems observability.
The reality for most companies is that once deployed, tracing systems get very little real use. Why?
Primarily because developers are used to logging and stats, but also because it is simply too hard to find relevant traces, especially when sampling, with existing tooling.
The end result is that a company spends a bunch of 💰 and time deploying and supporting something that does not have sufficient value proposition. This is why I recommend extreme caution today when considering bringing tracing to an org. Will the value be there?
As I said early on, I *do* think tracing data is crucial for understanding large systems. The issue is we haven't built a UI that can unlock that value for the average user.
Luckily, I see several startups that really seem to get this, and some of the mock-up UIs that I've seen are incredible, offering layered introspection of the system with tracing being but one of the detail views. I can't wait for these products.
Until then, if considering deploying tracing using the existing crop of products, I would pass. The value just isn't there.
