Thread by @jessitron

How to build reliable systems under unpredictable conditions

@MrBWilms is excited to be at #QConSF :-)

Normal testing is happy-path testing.

@MrBWilms #QConSf

Reliable: consistently good in quality or performance.
Reliable: able to be trusted.

@MrBWilms #QConSF

Note that he doesn’t say “perfect.” Good isn’t perfect, but it can be _real_.
And trust is not assuming perfection.

What are the conditions we are dealing with?
(8 fallacies of distributed computing)

the network is NOT reliable, secure, free, quick, unlimited, static, homogenous, or the product of one admin.

So our conditions are NOT predictable.

Unit testing is testing the bubbles. Not the connections, and especially not the unpredictable connections.

@MrBWilms #QConSF

Production is not a beautiful place.

@MrBWilms #QConSF

(I was too slow to catch the beautiful picture of fire that said “Production hates you”)

“My customer is not calling me."

This is not a strong metric.

Let’s be aware of production problems before that.

@MrBWilms #QConSF

The cycle of death: incident -> fix -> deploy -> something else went wrong

“Welcome to my life."

@MrBWilms #QConSF

we need

1. Testing
2. Resilience

Resilience is an outcome... a system’s ability to recover from a fault and maintain dependability

@MrBWilms #QConSF

… then @MrBWilms does a demo of steadybit, which is
- injecting a “blackhole attack” of taking out a service, then taking out DNS
- making the outcome legible: which services went down? Which ones are getting socket timeouts?

Time for some timeouts, circuit breakers, fallbacks, bulkheads for isolation...

and retries. “Be careful with retries.” You can cause more problems with a retry storm.

@MrBWilms #QConSF

“You have transformed an incident into an acceptance test.” by using chaos engineering to replay an incident.

using @steadybitHQ

There’s a cute graph of HTTP activity, and you can kind of see what kubernetes is doing. An insight into one of the universe’s unending mysteries

Test under real conditions in a safe environment.
Do it continuously. Like unit testing, integration testing, load testing.
Don’t be haunted by production.

@MrBWilms #QConSF

Congrats to @steadybitHQ for their first big US conference sponsorship.
Contact them for workshop on these principles in your company, and they can show how to

Express expectations about what your system should do under adverse conditions, and then verify that behavior.