Discover and read the best of Twitter Threads about #ChaosDay19

Most recents (16)

#ChaosDay19 Madaari Ordering for the Monkeys
#chaosday19 : [ed: Cool, Madaari is a joint work between eBay and Disorderly Labs, a combo of tech and academia. Love seeing these types of partnerships]
#chaosday19 testing faults can get really expensive
Read 15 tweets
#chaosday19 next up is disaster recovery at scale at Google by Parma Gopalan
#chaosday19 : DiRT started at Google back in 2006 [ed: might have gotten the date wrong] Awesome scenarios including what happens if a meteor hits and zombies start coming up from out of the ground.
#chaosday19 : Data center failures can occur for a multitude of reasons. Failures aren't just the loss of compute but also the issue of PoP vanishing.
Read 15 tweets
#chaosday19 : The Engineer in the Mirror with Randy Secrest Image
#chaosday19 : Talking about setting boundaries and space for work/life balance. Learning how to say no is critically important. Putting up guardrails and boundaries makes getting work done easier.
#chaosday19 : Recommendations: Don't lie to yourself. Prioritize what you value and what you want. Don't lead yourself by yourself - find likeminded people and start leading others.
Read 3 tweets
#chaosday19 : More CapitalOne conversation associated with Single Sign On services. We're hearing about chaos journey for this service.
#chaosday19 : A few elements to cover: operating point, MTTR, and human ops
#chaosday19 : Know your operating point (KYOP) is critical for high reliability organization. Based on Rasmussen Dynamic System model as well as Golden Signals (latency, rate, errors)
Read 8 tweets
#chaosday19 talking rough consensus. Image
#chaosday19 : talking about getting to be the 'first kid picked' for a sports team when it comes to tech. What's prohibiting us from being a better company? Lots of re-doing things over and over. Multiple implementations of monitoring tools for instance.
#chaosday19 : Comcast has modelled their process after ITF. "We reject kings, presidents and voting. We believe in rough concensus"
Read 9 tweets
#chaosday19 aaand super fast lightning talks. Moving on to Google SRE SLI/SLO and error budgets Image
#chaosday19 : Utilize SLOs (% of healthy reqs / all requests) for health of system. [ed: I like to think of SLOs + error budgets as a system's method of pulling a'andon cord.' Lean manufacturing posits workers pull a cord when critical issues arise. Here the sys does it for you]
#chaosday19 : Keeping SLAs more relaxed than your SLO is critical as you don't want your clients to hold you accountable to your own internal thresholds... otherwise you're going to have unhappy clients.
Read 3 tweets
#ChaosDay19 talk by Narendra Nalabothula, member of Capital One’s central chaos team. Exploring Chaos in a financial firm.
#chaosday19 : We talk about how failure is inevitable but often couple that with 'service failure is not an option!' This is largely due to the cost of the incident itself. [ed: we often discuss financial impact but should also consider brand/reputation impact too]
#chaosday19 : Chaos Engineering helps us think of failure as an absolute in production. Chaos implementation has a maturity path leading to automation via CI/CD pipeline.
Read 10 tweets
#ChaosDay19 @krisnova now up on Chaos Engineering.
#chaosday19 @krisnova : Managing chaos - how do we prep for something we can't prepare for? How do we get ready? Normally we manage it by building an "awesome" script to fix it all.
#chaosday19 @krisnova : How do we get to that script? Well something goes wrong. Time goes by. Humans get involved to repair, we triage, and then figure out the fix. Then this turns into a shell script after we go through the same process multiple times.
Read 13 tweets
#chaosday19 @aaronrinehart now talking about security precognition
#chaosday19 @aaronrinehart : We're going to be covering a LOT of stuff during this talk. Our systems have grown beyond our ability to grok them.
#chaosday19 @aaronrinehart case in point here’s Vizceral
Read 18 tweets
#chaosday19 Moving on to applying some chaos to Kube clusters! Image
#chaosday19 Glad to see the lightning talks are getting into some direct implementations over theory. Don't get me wrong I love theory but I really really like seeing the 'art of the possible' in action.
#chaosday19 Kube Monkey works by building a schedule of terminations based on opt-in status, chaos paramters and blacklist. Basically what to terminate and when. Terms are scheduled randomly across a configurable time window. Thankfully it does a sanity check right before term
Read 8 tweets
#chaosday19 Now moving onto a talk on @IstioMesh and how to introduce some chaos (and not just by installing it now :) )
#chaosday19 Istio (backed by Envoy) gives us some awesome out of the box options for failure injection including traffic aborts and request delays. You can apply these injections on percentage of request bases.
#chaosday19 In case you were curious, status code 418 is the code for "I am a teapot" developer.mozilla.org/en-US/docs/Web…
Read 4 tweets
#ChaosDay19 Time for Lightning talks! Starting with @beevek on Modern Traffic Stack Image
#chaosday19 @beevek is the co-founder and CEO of NS1
#chaosday19 @beevek : Starting to see clear patterns in modern traffic management stac. 3 Tiers: DNS/Global steering, Edge (edge cloud, cdn), and Origin (microservices)
Read 5 tweets
#ChaosDay19 @mipsytipsy up next with Closing the Loop on Chaos with Observability. [ed: anyone surprised that @mipsytipsy is discussing this?]
#chaosday19 @mipsytipsy is an engineer, founder of @honeycombio, big background in DB Reliability Engineering (she wrote a book)
#chaosday19 @mipsytipsy : "Chaos is a fancy marketing term for running tests later in the software dev lifecycle"
Read 27 tweets
#chaosday19 @gen_nja is getting started with his presentation “Black tie Chaos: Failing formally”
#chaosday19 @gen_nja : Providing a quick review of Auxon. Left the autonomous vehicle engineering and moved to complex systems at large.
#chaosday19 @gen_nja : Quick poll of who is a software engineer vs. who is a ChemE/MechE/etc "Everyone who is a software engineer, those other engineers don't think you're an engineer" *laughter and applause* "Who would trust a software engineer to build a skyscraper?"
Read 18 tweets
#chaosday19 @nora_js : Currently reviewing the first Chaos Engineering test the Apollo 1: Launch Rehearsal Test, a hugely public experiment that went wrong. A tragic event which changed spacecraft design forever.
#chaosday19 @nora_js has a huge background in Chaos Engineering, contributing to the Principles of Chaos book
#chaosday19 @nora_js : "We're going to be going through the 8 traps of Chaos Engineering today, a shoutout to @bergstrom_johan 's accident analysis"
Read 32 tweets
@caseyrosenthal and @nora_js getting ready to kick us off for Chaos Community Day Vol 4. I’ll be live tweeting all the talks today. We’ll see how this goes! #chaosday19 Image
#chaosday19 @caseyrosenthal "Tradition for Chaos Community Day is to start late and get very off track so we're keeping the tradition alive"
#chaosday19 @caseyrosenthal "Why are we here? The first Chaos Community day was the event that broke Chaos Engineering out beyond Netflix.... Netflix only hires senior people but Chaos Engineers didn't exist... so I started the events" [ed: somewhat paraphrased]
Read 5 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!