Discover and read the best of Twitter Threads about #ChaosDay19
Most recents (16)
#ChaosDay19 Madaari Ordering for the Monkeys 
#chaosday19 : [ed: Cool, Madaari is a joint work between eBay and Disorderly Labs, a combo of tech and academia. Love seeing these types of partnerships]
#chaosday19 testing faults can get really expensive
#chaosday19 next up is disaster recovery at scale at Google by Parma Gopalan
#chaosday19 : DiRT started at Google back in 2006 [ed: might have gotten the date wrong] Awesome scenarios including what happens if a meteor hits and zombies start coming up from out of the ground.
#chaosday19 : Data center failures can occur for a multitude of reasons. Failures aren't just the loss of compute but also the issue of PoP vanishing.
#chaosday19 : The Engineer in the Mirror with Randy Secrest
#chaosday19 : Talking about setting boundaries and space for work/life balance. Learning how to say no is critically important. Putting up guardrails and boundaries makes getting work done easier.
#chaosday19 : Recommendations: Don't lie to yourself. Prioritize what you value and what you want. Don't lead yourself by yourself - find likeminded people and start leading others.
#chaosday19 : More CapitalOne conversation associated with Single Sign On services. We're hearing about chaos journey for this service.
#chaosday19 : A few elements to cover: operating point, MTTR, and human ops
#chaosday19 : Know your operating point (KYOP) is critical for high reliability organization. Based on Rasmussen Dynamic System model as well as Golden Signals (latency, rate, errors)
#chaosday19 talking rough consensus.
#chaosday19 : talking about getting to be the 'first kid picked' for a sports team when it comes to tech. What's prohibiting us from being a better company? Lots of re-doing things over and over. Multiple implementations of monitoring tools for instance.
#chaosday19 : Comcast has modelled their process after ITF. "We reject kings, presidents and voting. We believe in rough concensus"
#chaosday19 aaand super fast lightning talks. Moving on to Google SRE SLI/SLO and error budgets
#chaosday19 : Utilize SLOs (% of healthy reqs / all requests) for health of system. [ed: I like to think of SLOs + error budgets as a system's method of pulling a'andon cord.' Lean manufacturing posits workers pull a cord when critical issues arise. Here the sys does it for you]
#chaosday19 : Keeping SLAs more relaxed than your SLO is critical as you don't want your clients to hold you accountable to your own internal thresholds... otherwise you're going to have unhappy clients.
#ChaosDay19 talk by Narendra Nalabothula, member of Capital One’s central chaos team. Exploring Chaos in a financial firm.
#chaosday19 : We talk about how failure is inevitable but often couple that with 'service failure is not an option!' This is largely due to the cost of the incident itself. [ed: we often discuss financial impact but should also consider brand/reputation impact too]
#chaosday19 : Chaos Engineering helps us think of failure as an absolute in production. Chaos implementation has a maturity path leading to automation via CI/CD pipeline.
#chaosday19 @krisnova : Managing chaos - how do we prep for something we can't prepare for? How do we get ready? Normally we manage it by building an "awesome" script to fix it all.
#chaosday19 @krisnova : How do we get to that script? Well something goes wrong. Time goes by. Humans get involved to repair, we triage, and then figure out the fix. Then this turns into a shell script after we go through the same process multiple times.
#chaosday19 @aaronrinehart : We're going to be covering a LOT of stuff during this talk. Our systems have grown beyond our ability to grok them.
#chaosday19 Moving on to applying some chaos to Kube clusters!
#chaosday19 Glad to see the lightning talks are getting into some direct implementations over theory. Don't get me wrong I love theory but I really really like seeing the 'art of the possible' in action.
#chaosday19 Kube Monkey works by building a schedule of terminations based on opt-in status, chaos paramters and blacklist. Basically what to terminate and when. Terms are scheduled randomly across a configurable time window. Thankfully it does a sanity check right before term
#chaosday19 Now moving onto a talk on @IstioMesh and how to introduce some chaos (and not just by installing it now :) )
#chaosday19 Istio (backed by Envoy) gives us some awesome out of the box options for failure injection including traffic aborts and request delays. You can apply these injections on percentage of request bases.
#chaosday19 In case you were curious, status code 418 is the code for "I am a teapot" developer.mozilla.org/en-US/docs/Web…
#chaosday19 @beevek is the co-founder and CEO of NS1
#chaosday19 @beevek : Starting to see clear patterns in modern traffic management stac. 3 Tiers: DNS/Global steering, Edge (edge cloud, cdn), and Origin (microservices)
#ChaosDay19 @mipsytipsy up next with Closing the Loop on Chaos with Observability. [ed: anyone surprised that @mipsytipsy is discussing this?]
#chaosday19 @mipsytipsy is an engineer, founder of @honeycombio, big background in DB Reliability Engineering (she wrote a book)
#chaosday19 @mipsytipsy : "Chaos is a fancy marketing term for running tests later in the software dev lifecycle"
#chaosday19 @gen_nja : Providing a quick review of Auxon. Left the autonomous vehicle engineering and moved to complex systems at large.
#chaosday19 @gen_nja : Quick poll of who is a software engineer vs. who is a ChemE/MechE/etc "Everyone who is a software engineer, those other engineers don't think you're an engineer" *laughter and applause* "Who would trust a software engineer to build a skyscraper?"
#chaosday19 @nora_js : Currently reviewing the first Chaos Engineering test the Apollo 1: Launch Rehearsal Test, a hugely public experiment that went wrong. A tragic event which changed spacecraft design forever.
#chaosday19 @nora_js has a huge background in Chaos Engineering, contributing to the Principles of Chaos book
#chaosday19 @nora_js : "We're going to be going through the 8 traps of Chaos Engineering today, a shoutout to @bergstrom_johan 's accident analysis"
@caseyrosenthal and @nora_js getting ready to kick us off for Chaos Community Day Vol 4. I’ll be live tweeting all the talks today. We’ll see how this goes! #chaosday19
#chaosday19 @caseyrosenthal "Tradition for Chaos Community Day is to start late and get very off track so we're keeping the tradition alive"
#chaosday19 @caseyrosenthal "Why are we here? The first Chaos Community day was the event that broke Chaos Engineering out beyond Netflix.... Netflix only hires senior people but Chaos Engineers didn't exist... so I started the events" [ed: somewhat paraphrased]
