Thread by @mipsytipsy

Oh man, do I ever have thoughts and feels on this. (Get yer popcorn.)

There are two major pathologies with tracing. First, it's daunting to roll out and make useful, and second, as you say, usage rarely spreads beyond that one or two experts per shop.

External Tweet loading...
If nothing shows, it may have been deleted
by @copyconstruct view original on Twitter

(I might add a third, which is its pricing model and sampling/retention promises and claims, but let's set that to the side as more of a business concern.)

So it's hard to set up and use, too daunting for most users to learn or remember how to use, there's no good way to find a particular trace.

And yet! Tracing companies can and do charge a mother fucking *ASSLOAD*... and people *pay it*.

Clearly there's value in it.

In fact, until honeycomb came along, tracing was the only game in town for debugging what we loosely call distributed systems, or microservices if you prefer.

Tracing had long been the only way you stood a chance debugging those systems, bc tracing describes events, not systems.

I say this over and over, but once again: metrics and aggregates describe the health of the SYSTEM. They have discarded all of the context of the event.

Metrics are useless for debugging events. You *must* sample instead of aggregate, you *must* preserve context in a struct.

A trace is just a series of events, with some headers to inform ordering etc. We only had to make a few minor tweaks to our storage engine to start supporting traces.

And we think we can uniquely help with both of those obstacles I listed. ☺️

For instrumentation: if you’ve already rolled it out, we jfw. But if you haven’t, and you’re loathe to embark on a massive refactor project before demonstrating value.. honeycomb users can just add a couple special headers, and voila you have trace data.

More interesting is the part about learning and adoption.

We have been obsessing over this since the beginning: “How do we bring everyone up to the level of every subject matter expert when debugging?”

It’s so, so cognitively taxing to learn and understand and debug a complicated problem. And everyone has to learn every problem from scratch? (& forget the details in a month)

But it’s so cognitively cheap to *share*, and to save it so that you can find it when you need it.

So honeycomb has always thought about these tools as things that are hard, with one local expert (if you’re lucky)

So naturally we want to help everyone piggyback off their knowledge. Look over their shoulder, so to speak, and stash away useful tricks for themselves.

We've only just begun to mine the learning and sharing space, but our intra-company adoption rates have always been absurdly strong. Despite honeycomb being roughly as complex and hard as tracing.

So I wouldn't write tracing off just yet. The value is there, but the tooling has been hostile.

In other words, if you want to roll out tracing and have all your engineers use it, I recommend giving honeycomb+tracing a try. 🐝

the end! (Cc @copyconstruct)