Thread by @mipsytipsy

"Metrics are just like a slap in the face. You see a metric, it's like a slap -- something happened! You go to find out what it was, you have no idea. It's just a number in the middle of nowhere" @GianArb throwing A+ shade #jazoon

"Collection is hard... because you don't know in advance what you're looking for. If you did, this would be easy" @GianArb #jazoon

“We are not in production yet, but we have an environment called production and we run it as though we are in production” @GianArb talking about baking code and processes continuously #jazoon

Every single pull request must include instrumentation that answers the question, how will I know if this isn't working? [ed: Observability-Driven Development!] @GianArb #jazoon

Measure how fast and how well you write and ship code -- observability for your teams! @GianArb #jazoon

A service isn't ready to ship until it has a /metrics endpoint. Their service isn't even live yet but has >300 metrics. They use prometheus, although it is problematic due to pre-aggregation and impossibility of point-in-time [ed: yes. inherent problem with metrics 😑]

Monitor your monitoring infrastructure. Also, colocate it as far away from production and as decoupled as possible from prod. Consider outsourcing to a provider for this reason alone. @GianArb #jazoon [ed: it's all fun and games until both prod AND monitoring infra go down]

Question from audience about retention and expense. Answer: store everything for a short time, then downsample intelligently on less critical details. @GianArb #jazoon [ed: Honeycomb has a fast query window for this purpose, after which it ages out to s3]

Question: should we collect metrics for things we don't alert on? @GianArb yes. [ed: HELL YES THIS IS INSANITY. You should gather *hundreds of times more* metrics than you alert on! 😱 Also, ed'ing like @lizthegrey is super fun!] #jazoon

Ed note, 300-ish metrics for a pre-prod service sounds just about right. We observe rich, fully baked production services often settle around 400-500 dimensions.

(In honeycomb terms this would be a single event per request with 400-500 dimensions, not 400-500 isolated metrics.)