Thread by @tleam

#ChaosDay19 Madaari Ordering for the Monkeys

#chaosday19 : [ed: Cool, Madaari is a joint work between eBay and Disorderly Labs, a combo of tech and academia. Love seeing these types of partnerships]

#chaosday19 testing faults can get really expensive

#ChaosDay19 history of chaos. First is the random model, think Chaos Monkey. Next is guides fault injection. Both valuable neither scalable

#chaosday19 : Fault Tolerance = Redundancy. If a system is fault tolerant it is necessarily redundant, either via redundancy of system artifacts or behavior. [ed: not sure I fully buy in. Fault tolerance where completely alternative or 'acceptable dropping' doesn't seem to fit]

#chaosday19 LDFI recipe: work backwards, lineage trace, convert lineage to CNF, repeat

#chaosday19

#chaosday19 : pruning experiments based on LDFI utilizes expert intuition. If you can prove that a direct dependency failure is managed by a higher node there's no need to test additional downstream dependency failures for a particular context as they are handled already

#chaosday19 : Other prioritization measures include using 'known frequent failure modes' as well as the history of failure and deployments for a particular service. A service recently release is more likely to cause an issue than one that has been stable for some time.

#chaosday19 : to validate interaction replay is important as well, using E2E tests with a tool like Selenium can help.

#chaosday19 : Using LDFI in the enterprise - Traces/Structured logs can be mapped to call-graphs and encoded for LDFI. Then a fault suggestion should be provided to use with your FIT Tool.

#chaosday16 I’m finding this talk to be absolutely fascinating in providing value for more efficiently executing chaos experiments. Simultaneously, it’s quite the talk to mentally keep up with at the end of the day.

#chaosday19 : While Madaari primarily focuses on microservices the future will certainly need to encompass other elements such as dtabases, containers, service mesh, etc.

#chaosday19 : The 3 W's for Fault Injection:
- What to inject? (type of fault to inject)
- Where to inject? (target component)
- When to inject?

@threadreaderapp unroll this please