Thread by @1z0b31

#observability has three pipes (not pillars!)

As humans, we can't (and don't want to!) investigate each individual trace. But how we do know which traces matter? Which are representative?

Root cause analysis is one thing when you have low cardinality...

But the real world usually demands high cardinality

Statistical analysis is your friend and ally!

What stands out about the slowest (eg p99) requests? What distinguishes them from the rest?

Imagine catching this through a metrics dashboard. It's not realistic!

Let the computer find the critical path for you! Make the machines for the work 🤖🤖🤖

If you shrink the duration of the red bar in aggregate, you improve the latency of all the requests

*Where* is the first step, but it's usually the easiest part.

*Why* is both harder and more valuable... and often requires human intervention to determine.

Span tags to the rescue!

Who's waiting in the critical path? What are they waiting on?

Donut baking is very distributed 🍩🍩🍩

Aggregate analysis confirms that the mutexes are the problem

Mutex analysis (!!!)

(@dymxzvf deliberately added more cinnamon demand + made chocolate slower)

Immediate 💡: use different mutexes for different toppings (ie paths)

Single traces are just the start of the journey. Aggregate tracing is where the true value lies ✨✨✨

🔭 @opentelemetry 🔭 will make this easier

(and @opentracing and @opencensusio in the mean time)

Why does it take so long to require the lock?! What's holding up all the traffic?

It takes a bird's eye view to analyze the situation