, 25 tweets, 3 min read Read on Twitter

What causes disasters, like Chernobyl, Deepwater, Challenger, & Titanic?

-Perverse incentives <— cause
-Corprorate-Culture, deception, ... <— mechanism, not cause

-How beliefs, culture work
-How to think bout causality in such contexts

(Details ⬇)
What perverse incentives?

- Rewarded for haste
- Rewarded for cutting costs
- Insufficiently penalized for causing harming

➥ Incentivizes risky behaviors

-Titanic trying for record breaking maiden voyage.
-Chernobyl trying behind schedule, trying to get test in last night possible.
-Deepwater costs 500k/day to operate.
-Challenger trying to quadruple rate, so can get goods to space w/ reusable shuttles.

-Titanic used cheap iron rivets, more likely to pop when hull plates flexxed
-Chernobyl used cost-saving boron-tipped control rods, known to increase reactivity instead of decrease, momentarily when inserted.
-Deepwater avoided expense of recommended cement bond-log test.

-BP had history of being put on probation or slapped on wrist, for disasters caused by negligence+fraudulent cover-up.
-USSR not known for environmental care. Or caring bout lives of citizens.
-Morton-Thiokol, subcontractor for NASA worried bout losing contract. Not deaths.
What happened:

-Risks under-estimated
-Safety mechanisms curtailed
-Warnings unheaded

➥ Disasters occurred

-Titanic “unsinkable.” (Truth: water containment effective if only few sections breached.)
-Pressure test showed Deepwater well secure (Truth: pressure tests ambiguous.)
-Challenger O-Rings not proven to fail at low temperature. (Sure, but never tested.)

Primary O-rings failed so consistently, secondary O-rings, meant as backup, were relied on as primary. Also: Morton-Thiokol engineers recommended against launch due to weather. Managers overrode.

Manager at Chernobyl ignored computer & engineer suggestions to halt test, relying on shutdown button for emergency. But button known to backfire sometimes, due to boron-tips.

Titanic ignored 6 iceberg warnings. Continuing at full speed. Despite poor visibility and no binoculars for lookouts. Also removed many lifeboats before voyage, to improve first class view. Never trained staff for evacuation. And lowered bulkheads so less effective.
How were risks under-estimated?

-Ambiguous signals talked down
-Bad signals weren’t relayed

-Chernobyl employees not told shutdown button known to backfire. Or other known problems w/ rbmk reactor.
-titanic staff unaware ship sinkable, insufficient life boats. Even as sinking.
-NASA hierarchy never informed of O-ring weather concern.
What did “corporate-culture” look like?

-Perverse incentives “passed on”
-Information flow impeded

(I.e. lower level employees incentivized to cut costs, ignore risks. Not informed of known risks. Precluded from conveying red flags.)

BP employees fired for raising concerns. Promoted for running shady safety checks, often w/o even visiting site.

NASA made clear to Morton-Thiokol contract up to renogiation. Time table essential. Showed frustratiin when engineers mentioned weather concerns.
My claim:

Bad corporate-culture+biased risk assessment.
<— consequence of perverse incentives.

- Management has incentive to create this corporate-culture. And to generate these risk assessments.

- B/c helps ensures others act according to their interests. And can better justify, persuade them to do so.
Who cares?

1) understanding causality tells you what need to change to fix problem.

2) if miss causal structure, miss key insights on how beliefs, corporate culture work.

If misidentify corporate-culture, or mistaken beliefs, as ultimate cause, tempted to adjust those w/o addressing source. W/ minimal effect.

If address perverse incentives, corporate-culture+mistaken beliefs will follow.
Illustrates common way beliefs influenced by incentives:

- beliefs biased in way that helps justify actions, persuade others to enact.
- e.g. perverse incentives that promote risk —> under-estimate risk.
-Motivated Resoning, restricted info flow <— means of achieving above.

-*very* different from beliefs biased by “what want to be true.” As social psych claims.

(NOONE wants disaster to occur. But that doesn’t explain why specifically those w/ perverse incentives to take risk ESPECIALLY prone to under-estimate.)
Note also:

-*very* different from standard econ approach of presuming beliefs determined by priors+information.

(Leaves NO ROOM for perverse incentives to come into play. Beliefs unaffected by what need to justify or persuade.)

Another key recurring feature of how incentives influence beliefs and behavior:

They often get “passed on.”

(Just like how climate denial helps big oil justify continued polluting. And they “pass on” these perverse beliefs to others they influence <—all republicans.)

Corporate-culture, like beliefs, don’t come from nowhere.

And don’t get far if just treat as exogenous. Or as resulting from some kind of inherent flaw in humanity or the mind.

Comes from incentive structure.
(Namely: Perverse incentive to take risk—> culture that promotes risky behaviors, and precludes info transmission re known risk factors.)
True of all “culture.”

Not exogenous force. Shaped by incentives.

(E.g. incentives of leaders? E.g. nationalism promoted by those who benefit from you sacrificing for their group? Like heads of states leading masses to slaughter in ww1, promoting nationalististic norms?)

Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Moshe Hoffman
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!