Yes, yes. Sorry. I did not mean "screw TDD, find every bug in production and ONLY production". That would be too obviously ridiculous, or so I thought ☺️ but many did read it that way.
It is cheap and easy to find known unknowns, so we should check for them CONSTANTLY.
It is cheap and easy to find known unknowns, so we should check for them CONSTANTLY.
So much of running production systems comes down to your skill at turning unknown-unknowns into known-unknowns with some consistency.
Every unknown-unknown is an engineering problem. It requires creativity, novel thinking, and an open-ended amount of time to resolve.
Every known-unknown is a support problem. A pattern matching problem, an index lookup problem.
Every known-unknown is a support problem. A pattern matching problem, an index lookup problem.
Unknown-unknowns are incredibly, catastrophically expensive to debug, especially for teams who do it rarely +/have not built their systems with an eye to doing it well.
Unless you are a world class SRE team, your team can handle *maybe* one per week and still ship some code.
Unless you are a world class SRE team, your team can handle *maybe* one per week and still ship some code.
More than that, and your roadmap would grind to a halt. Or you'd do what everyone else does and cut corners. Many, many corners.
If it recovered on its own somehow, you'd shrug uneasily and get back to work, never actually knowing what happened.
If it recovered on its own somehow, you'd shrug uneasily and get back to work, never actually knowing what happened.
You'd file a bunch of tasks on how to fix it so the problem wouldn't happen again, and let the tasks rot away in jira forever.
You'd document the symptoms and rely on people's memory, instead of instrumenting the system to be clearer and saner.
You'd document the symptoms and rely on people's memory, instead of instrumenting the system to be clearer and saner.
You'd talk, again, about adding honeycomb. (Maybe next year, or whenever things are "less busy".)
And your system would sink a little further into the bog of unintelligibility every time. But I get it -- I've been there. Doing things well takes time.
And your system would sink a little further into the bog of unintelligibility every time. But I get it -- I've been there. Doing things well takes time.
That's why it's of such shimmering importance that, when you turn an unknown-unknown into a known-unknown, you at the very least DO NOT LOSE IT.
Write tests to check for software regressions. Write monitoring checks to check for system regressions. Run them a lot.
Write tests to check for software regressions. Write monitoring checks to check for system regressions. Run them a lot.
