,
11 tweets,
6 min read
Read on Twitter
CHAOS! Breaking your systems to make them unbreakable @gitbisect #TrajectoryConf
The advantage of cloud is being able to easily spin up new instances. But it is NOT a guarantee against failure. @gitbisect #TrajectoryConf
Game days–Timing: 30 minutes of planning, 50 minutes of playing, 10 minutes of reporting. @gitbisect #TrajectoryConf
Game days–The team: 1 subject matter expert, 1 SRE, sometimes 1 junior engineer/developer. Creates a great learning environment and additional perspectives, can ask why. @gitbisect #TrajectoryConf
Game days–planning: schedule it, don't choose complex tasks, start easy, create a hypothesis, write down expectations and what to do when things go wrong, document and share the plan. @gitbisect #TrajectoryConf
Game day–game time: Start in staging, move to production (off-peak), move to production (primetime); announce and maintain discussion in group chat; monitor for outages; run tests and take notes. @gitbisect #TrajectoryConf
Game day–after: Create tickets to track issues uncovered, write summary and lessons learned, email, celebrate. @gitbisect #TrajectoryConf
Celebrate everything. Even if you broke everything. Celebrate learning–that is what matters. @gitbisect #TrajectoryConf
Game day–game level complexity: start with terminating service–>block access to 1 dependency–>block access to all dependencies->terminate the host->degrade the environment->spike traffic->terminate the region cloud. @gitbisect #TrajectoryConf
@gitbisect is now tempting fate with a live chaos experiment. #TrajectoryConf
Game day–In summary: Share. Plan. Have fun. @gitbisect #TrajectoryConf
