Thread by @jessitron

This morning at #QConLondon , @glenathan tells a story of an experiment at Circle CI: for this greenfield project, let’s have NO LOGGING.

Instead of logs, events! that make traces!

“Events are good logs.”
They’re data-first instead of human-first; they wrap units of work with durations; they can be explored multiple ways

@glenathan #QConLondon

They found best practices like: wrap each unit of work; report errors in a standard field; use span names that are specific enough to tell you what’s happening but general enough for useful grouping.

~ @glenathan #QConLondon

Then they tuned the span creations so that there wasn’t a bunch of duplicate-looking ones. Instead of repetitive spans, add more attributes.
“Add every identifier you have in your system” to help find problems.

@glenathan #QConLondon

For reassurance, they sent the events to their existing Elasticsearch as well as to @honeycombio .
But nobody looked at them there, so they turned it off later.

This part is cool: they turned the spans into formatted stdout lines for local dev.

And they emitted some of their span attributes to statsd for metrics, for the constantly-updating dashboards.
This gave them dashboards about what the service was doing.

If you look at the “three pillars” of observability from the top,
it’s all the same events.

@glenathan #QConLondon

Thinking trace-first proved useful.
Putting all this thought into a new way of doing things was meh —like, it was better, but not revolutionary.
(darn)