Thread by @el_bhs

For observability, scale is a red herring: what really matters is *depth*.

And once you start thinking about “deep systems,” you realize why conventional observability dogma is nonsensical. And also why tracing will become the foundation for effective observability.

Thread: /0

First, let’s talk about scale and why it’s the wrong way to think about system complexity.

“Scale” refers to throughput, cost, developer headcount, or some other linear attribute of a system. /1

You can deploy a simple function that resizes images without making any system calls, wrap it in an RPC handler, and scale that to your heart’s content: observability will be straightforward (though justifying your compute bill may be a different story!). /2

When organizations move to microservices, though, something different happens: the systems get *deep*. As in, ordinary transactions pass through many layers of microservices before returning a response to the user. /3

Depth is often confused with scale since it correlates with headcount growth and transaction volume, but our failure to understand the structure of deep systems explains our failure to properly observe them. /4

In a “deep system,” most transactions make *many* hops across service boundaries – hence “depth.” And every hop is a new opportunity for things to go sideways. /5

As I wrote in a previous post (lightstep.com/blog/three-pil…) the “three pillars of observability” is basically garbage. Metrics, logs, and traces are the data, not the solution. /6

Only distributed traces can provide the context we need to rank and filter the overwhelming and undifferentiated streams of metric data. /7

And modern distributed tracing can make well-informed sampling decisions, overcoming the crippling dollar cost of conventional logging in deep systems. /8

So, when ordinary systems grow to become deep systems, tracing data is not only *mandatory*, but in fact must form the backbone of an effective observability system. /9