Thread by @berkun

1. I keep thinking about system design and how we're all getting big lessons way too fast.

We don't recognize how many things are interconnected, even when the first fails, and the second, until the cascade is undeniable even by the skeptics. And then it's too late to catch up.

2. Systems theory sounds wonky until you face a situation where you can't explain what's happening in conventional, single action/response oriented ways.

If doing A suddenly stops giving you B, it means there are other factors in why A worked at all. You didn't see C,D,E...

3. Things like feedback loops and exponential effects defy ordinary daily workaday logic. Most people don't believe in it until they see it.

“We have created trouble for ourselves in organizations by confusing control with order.”
– Margaret J. Wheatley

4. Our narrative bias wants us to believe there is a single point of failure when terrible things happen, but that's rare.

Mostly it's interdependent choices where one or more ppl acted in ignorance of how one choice impacted the others.

Major failures share patterns.

5. It's when disaster happens people realize how much more complex things are than they thought. It's a shock.

Mostly the thing worked! It was dependable! But then ignorantly pushing it beyond its (hidden) limits blows it up and it's a surprise.

We're all surprised right now.

6. Thinking like a designer some simple systems give clear feedback when it's about to fail. Car odometers all have a red-zone. Power is great! But there are limits.

But the more complex the system the harder it is to see one simple measure of danger. Easier to ignore or deny.

7. It's the people with the most knowledge of the system, and the history of systems like it, that are the most reliable judges of when we're approaching redlines.

But if the people at the wheel have idealistic optimism, and are prone to ignorance, experts are easy to ignore.

8. In complex systems that fail there's usually someone who called out the danger but was ignored. Why? Their boss thought the system was simpler than it was.

"breakdowns of human Systems are attributable to misunderstandings—in brief, to failures of communication." - John Gall

9. Wisdom suggests complex systems are built with redundancy. With insurance. You bet that there are things you don't know. It's worth it because failure would be tragic.

When Roebling built the Brooklyn Bridge he made it 6x stronger than it needed. To insure system resilience.

10. But over time it's easy to see redundancy as inefficiency, regulation as bloat/bureaucracy. Why 6x when "1x done right" should do?

Enter the Tacoma Narrows bridge. Designed to be "innovative", light and inexpensive. Which tore itself apart.

11. And for Roebling, it turned out one of his cable suppliers was corrupt. And defective cable was installed, and is still part of the bridge to this day. His 6x was reduced to 4x.

Complex systems have many surprising ways to fail. Unless you design insurance against them.

Whoops - not an odometer you idiot! It's a Tachometer. Even I know that. Grrrr.