Thread by @tleam

#chaosday19 @nora_js : Currently reviewing the first Chaos Engineering test the Apollo 1: Launch Rehearsal Test, a hugely public experiment that went wrong. A tragic event which changed spacecraft design forever.

#chaosday19 @nora_js has a huge background in Chaos Engineering, contributing to the Principles of Chaos book

#chaosday19 @nora_js : "We're going to be going through the 8 traps of Chaos Engineering today, a shoutout to @bergstrom_johan 's accident analysis"

#chaosday19 @nora_js: "Trap 1: Counting vulnerabilities is not a success measure for Chaos Engineering."

#chaosday19 @nora_js : "What does the vulnerability count actually mean? It's about understanding them and their context"

#chaosday19 @nora_js : Currently referencing the book "How to lie with statistics" by Darrell Huff. [ed: Fantastic and quick read that's super applicable today even though it was published decades ago.]

#chaosday19 @nora_js : "Purpose isn't to find vulnerabilities through tooling, the goal is to push forward on a journey of resilience through the vulnerabilities we find" [ed: 100% agreed. I find Chaos Engineering to be an awesome tool for refining our mental model of the system]

#chaosday19 @nora_js : "Looking at what went right in a chaos experiment helps us understand what went wrong. We don't do this nearly enough. What if you find out that a single employee knew to check a single dashboard that no one else thought of? That's valuable!"

#chaosday19 @nora_js : "A story about gameday preparation issues - 2 of 11 engineers invited to gameday prep, this was ideally to save time. We didn't run the gameday. Gamedays and prep for the gameday prep is essential so you don't miss out on mental models of all involved"

#chaosday19 @nora_js : "Trap 2 in the Apollo 1 Launch Experiment was that not everyone that was essential was involved in the prep."

#chaosday19 @nora_js : "Trap 4 - Chaos Engineers shoudl be rule enforcers. Unfortunately some CEs find themselves becoming responsible for systems they don't own."

#chaosday19 @nora_js : "I've seen situations where 'we ended up piling action items to a team with a JIRA backlog that was 7 pages long' - don't do this! We should be providing context"

#chaosday19 @nora_js : "Trap 4.5 - You should fix the vulnerabilities you find and if you can automate the fixes"

#chaosday19 @nora_js : "We ran into this issue while running experiments with Monocole at Netflix. This was new information for teams. Almost every time we'd receive requests 'Hey you know this timeout is broken, could you fix it?'"

#chaosday19 @nora_js : "A machine should not tell us where and how to implement timeouts but it can give us information as to why their current state might be wrong"

#chaosday19 @nora_js : "Trap 5 - IT's not "real" Chaos Engineering unless you move beyond gamedays and experimenting in sandbox environments" [ed: 100% agreed, seen this trap numerous times and there's tons of value in non-production experiments. There's value at every syslevel]

#chaosday19 @nora_js : "You can get more value going through the process of automating experiments than you do actually automating them"

#chaosday19 @nora_js : "Trap 6 - the most important part of Chaos Engineering is running the experiment. The before and after experiment are actually the highest value elements of the experiment process itself"

#chaosday19 @nora_js : "All 3 phases (before, during, after) deserve equal attention"

#chaosday19 @nora_js : "Before the experiment - understand that all mental models of a system are wrong but discussing them amongst a team provides significant value"

#chaosday19 @nora_js : Unless we share mental models we make assumptions (often perfectly reasonable) about how a system operates as well as how we think others think the system operates. [ed: our inherent bias is to assume everyone has the same mental model as us]

#chaosday19 @nora_js : " Trap7 you can worry about safety later, the important part is creating the chaos" [ed: Narrator: no it isn't]

#chaosday19 @nora_js : "For the Apollo 1 example the hatch door was built with safety in mind but during spaceflight it became stuck so it couldn't open up"

#chaosday19 @nora_js : "Creating chaos is easy. Thinking about safety is *hard*"

#chaosday19 @nora_js : "Trap 8 - you can do chaos engineering without having an understanding of the system you are experimenting on." [ed: this is a great way to fall into Trap 7... if you don't know the system you can't even start to think about safety measures]

#chaosday19 @nora_js : "This gets into why it's important to have observability"

#chaosday19 @nora_js : "Trap 8.5 - It's ok to be a little lenient about 'steady state'"

#chaosday19 @nora_js : "For the Apollo 1 Launch there were a number of signals that things weren't right but they launched anyway"

#chaosday19 @nora_js : "Trap 3 - There is a prescriptive formula for doing Chaos Engineering. Which is why this slide is out of order!"

#chaosday19 @nora_js : "Can't have a prescriptive formula for the unpredictable" [ed: really like this quote, I'll definitely be using and attributing this quite a few times in the not too distant future]

#chaosday19 @nora_js - and that's a wrap /fin

@threadreaderapp unroll please!