Thread by @PierreVincent

First talk of the day on the #QConLondon production track, by @glenathan and a challenge: can we build observable services without logs?

"We needed to build a new service in Go, without our usual existing scaffolding in Clojure... That led to some bikeshedding but also gave a chance for experimentation!" @glenathan #QConLondon

"Before this, we spent a lot of money to know what our applications were doing in production"

Giving up logs, for an event-first approach. "Events are just good logs" @glenathan #QConLondon

Totally agree with @glenathan: it's ok to "clutter" your code with telemetry code, because it's right there in front of you, and encourages to do it consistently. Other approach like aspect oriented can hide it away, and gets forgotten.
Telemetry first! #QConLondon

It's not only about high-cardinality (eg user id...) but expressing more about units of work with good span names, gives a solid high level view of traces. @glenathan #QConLondon

... But still do the high cardinality stuff! Throw all information, context and ids available at that stage (it's doesn't cost anything extra in a volume-based pricing!) @glenathan #QConLondon

Instrumenting with trace first doesn't mean we can't have logs, especially when developing locally or running tests. With a bit of magic on pretty printing all of the json events, and outputting events during failed tests. @glenathan #QConLondon

So that's logs and traces from events. Next is metrics...which can also be derived from the events, with the same source of truth. @glenathan #QConLondon

So the 3 pillars are actually all the same thing, just from a different angle. @glenathan #QConLondon