Thread by @mipsytipsy

i keep forgetting to link to this longer piece. it tells the tale of how we tested a script to *terminate our entire infra and replace it with a newer OS* in prod.

"spend less time on what doesn't matter, more on what does. Test In Prod." techbeacon.com/app-dev-testin…

but hey: the biggest reason "Don't Test in Prod" was a cardinal principle for years was because all you had was monitoring tools and aggregates, and you legit couldn't tell at a hi-res granularity what the subtle effects of your tests were.

if you're flying blind, *don't do it.*

observability -- the way i define it, which is NOT how every other vendor out there defines it -- requires the ability to slice and dice down to raw events, and group by any combination of multiple high cardinality dimensions.

so you can see what the loving hell you're doing.

time series intervals, averages of averages, fixed schemas, indexes, this shit is anathema to peering closely at the ripple effects of tiny changes to tiny parts of your systems.

DO NOT be all "oh charity says test in prod is fine" if you don't have the o11y to check yourself.

observability is the missing link to most recent trends in systems: chaos engineering, testing in production ... everything post microservices demands it.

call it observability or something else, i dont care, but get clear on the definition or you WILL hurt yourself.