Thread by @mipsytipsy

I'm getting pushback here, but I'm going to double down.

The vendor line seems to be "if you have services in prod, you're ready for chaos engineering". Maybe in the sense that you can do it, and possibly derive benefit, but not in the sense that it's the best use of your time.

https://twitter.com/mipsytipsy/status/1218960242454581248

Why? If you are still dashing from fire to fire, not budgeting for reliability work, then whatever you learn from your chaos experiments is going to get piled on the ever-growing todo list.

Action items from real outages will always trump those from self-inflicted experiments.

So until you've gotten a handle on your real outages, there isn't much point ginning up more work for yourselves that everybody knows won't get done.

And why observability? (Btw, this is easier by far. You can install honeycomb beeline libraries on your edge and services in ... however long it takes you to run a deploy.)

Because I'm sick of seeing people find their broken shit days, weeks, or months later.

If you only have monitoring, you don't have the ability to slice and dice by every high cardinality dimension -- host, process, instance, container, version -- to find the littered trail of injected chaos.

If you can't see what you're doing, don't do it. Please.

There is one important caveat to this. While there is no technical justification for doing chaos engineering before you have observability (in the technical sense) and budget your reliability, there may be a political justification.

I am not qualified to comment there.