,
18 tweets,
6 min read
Read on Twitter
#observability has three pipes (not pillars!) 
As humans, we can't (and don't want to!) investigate each individual trace. But how we do know which traces matter? Which are representative?
Root cause analysis is one thing when you have low cardinality...
But the real world usually demands high cardinality
Statistical analysis is your friend and ally!
What stands out about the slowest (eg p99) requests? What distinguishes them from the rest?
Imagine catching this through a metrics dashboard. It's not realistic!
Let the computer find the critical path for you! Make the machines for the work 🤖🤖🤖
If you shrink the duration of the red bar in aggregate, you improve the latency of all the requests
If you shrink the duration of the red bar in aggregate, you improve the latency of all the requests
*Where* is the first step, but it's usually the easiest part.
*Why* is both harder and more valuable... and often requires human intervention to determine.
*Why* is both harder and more valuable... and often requires human intervention to determine.
Span tags to the rescue!
Who's waiting in the critical path? What are they waiting on?
Donut baking is very distributed 🍩🍩🍩
Aggregate analysis confirms that the mutexes are the problem
Immediate 💡: use different mutexes for different toppings (ie paths)
Single traces are just the start of the journey. Aggregate tracing is where the true value lies ✨✨✨
Why does it take so long to require the lock?! What's holding up all the traffic?
It takes a bird's eye view to analyze the situation
It takes a bird's eye view to analyze the situation
