Thread by @tleam

#ChaosDay19 @mipsytipsy up next with Closing the Loop on Chaos with Observability. [ed: anyone surprised that @mipsytipsy is discussing this?]

#chaosday19 @mipsytipsy is an engineer, founder of @honeycombio, big background in DB Reliability Engineering (she wrote a book)

#chaosday19 @mipsytipsy : "Chaos is a fancy marketing term for running tests later in the software dev lifecycle"

#chaosday19 @mipsytipsy : "I think this is really just 'lets test in production'"

#chaosday19 @mipsytipsy : You can test both in prod and nonprod! You can test the... out of it in non-prod. [ed: meme game is strong here. Lots of parallels to @nora_js traps from earlier today]

#chaosday19 @mipsytipsy : Deploying code is not a binary switch, deploying code is a process of increasing confidence in your code.

#chaosday19 @mipsytipsy : "Deploying is turning on the oven to start baking your production cookies" [ed: hopefully you're baking in small batches at first!]

#chaosday19 @mipsytipsy : Why does our mental model fall apart after a series of deployments and flags

#chaosday19 @mipsytipsy : "Largely due to increasing systems complexity. How do we know this is true? Because I drew a graph to show that it is"

#ChaosDay19 @mipsytipsy differences in complexity of systems within a decade

#chaosday19 @mipsytipsy : "We are all distributed systems engineers now. Our predictions of how the system will perform is always a lie... Whenever something truly unpredictable happens it's all hands on deck to try and figure out what's going on (normally networking)"

#chaosday19 @mipsytipsy : "Distributed systems are particularly hostile to being cloned or imitated (or monitored). Is anyone going to spin up a clone of Facebook to test it? Diminishing returns fall off dramatically"

#chaosday19 @mipsytipsy : "Distributed systems have an infinitely long list of almost-impossible failure scenarios that make staging environments *particularly* worthless" [ed: systems are in constant state of unique failure! Not sure I agree with 'particularly worthless' tho]

#chaosday19 @mipsytipsy : "Without observability you don't have chaos engineering, just chaos. If you can't see what's going on, you are not tall enough to ride this ride" [ed: preach!]

#chaosday19 @mipsytipsy : "Monitoring systems have not changed significantly in 20 years - @grepory "

#chaosday19 @mipsytipsy - getting into the definition of Observability from control theory "a measure of how well internal states just by asking questions from the outside" [ed: based on above/below line I'd argue this is the only way we have *any* understanding of the sys]

#chaosday19 @mipsytipsy : "You have an observable system when your team can quickly and reliably track down any new problem with no prior knowledge or scar tissue that you've built from historical incidents"

#chaosday19 @mipsytipsy : "#o11y takes the perspective of the sofware"

#chaosday19 @mipsytipsy : "You *have to test in production* This means being able to run tests that you do not know the answer to" [ed: once we admit that we're in constant flux, I'd argue that most if not all 'touches' of production are in fact... a test]

#chaosday19 @mipsytipsy : "Every time an issue occurs you could technically add an alert/monitor for it. If it happens again you can catch it!" [ed: but then you have a tangled pile of alerts and monitors that rot away over time and you'll eventually deal with alert fatigue]

#chaosday19 @mipsytipsy : "Context is key when debugging things. Events provide more context than metrics including high cardinality (must have!)"

#chaosday19 @mipsytipsy : Non-interactive dashboards are garbage... maybe have one or two on a wall. These are just artifacts of past failures and don't enable debugging.

#chaosday19 @mipsytipsy : "Debugging requires getting to individual requests" Aggregates are a kiss of death in this space "Once you smush everything into an aggregate you can't unsmush it"

#chaosday19 @mipsytipsy : Giving developers observability tooling they need to become fluent in production and to debug their own systems is critical.

#chaosday19 @mipsytipsy : "We must watch our code run in production. If you don't watch your code on fire you won't know what you're doing"

#chaosday19 @mipsytipsy : "Tonight we test in prod!" /fin

@threadreaderapp unroll