Thread by @Moshe_Hoffman

Thread:

What causes disasters, like Chernobyl, Deepwater, Challenger, & Titanic?

Claim:
-Perverse incentives <— cause
-Corprorate-Culture, deception, ... <— mechanism, not cause

Lessons:
-How beliefs, culture work
-How to think bout causality in such contexts

(Details ⬇)

What perverse incentives?

- Rewarded for haste
- Rewarded for cutting costs
- Insufficiently penalized for causing harming

➥ Incentivizes risky behaviors

E.g.

-Titanic trying for record breaking maiden voyage.
-Chernobyl trying behind schedule, trying to get test in last night possible.
-Deepwater costs 500k/day to operate.
-Challenger trying to quadruple rate, so can get goods to space w/ reusable shuttles.

E.g.

-Titanic used cheap iron rivets, more likely to pop when hull plates flexxed
-Chernobyl used cost-saving boron-tipped control rods, known to increase reactivity instead of decrease, momentarily when inserted.
-Deepwater avoided expense of recommended cement bond-log test.

E.g.

-BP had history of being put on probation or slapped on wrist, for disasters caused by negligence+fraudulent cover-up.
-USSR not known for environmental care. Or caring bout lives of citizens.
-Morton-Thiokol, subcontractor for NASA worried bout losing contract. Not deaths.

What happened:

-Risks under-estimated
-Safety mechanisms curtailed
-Warnings unheaded

➥ Disasters occurred

E.g.

-Titanic “unsinkable.” (Truth: water containment effective if only few sections breached.)
-Pressure test showed Deepwater well secure (Truth: pressure tests ambiguous.)
-Challenger O-Rings not proven to fail at low temperature. (Sure, but never tested.)

E.g.

Primary O-rings failed so consistently, secondary O-rings, meant as backup, were relied on as primary. Also: Morton-Thiokol engineers recommended against launch due to weather. Managers overrode.

E.g.

Manager at Chernobyl ignored computer & engineer suggestions to halt test, relying on shutdown button for emergency. But button known to backfire sometimes, due to boron-tips.

E.g.

Titanic ignored 6 iceberg warnings. Continuing at full speed. Despite poor visibility and no binoculars for lookouts. Also removed many lifeboats before voyage, to improve first class view. Never trained staff for evacuation. And lowered bulkheads so less effective.

How were risks under-estimated?

-Ambiguous signals talked down
-Bad signals weren’t relayed

E.g.

-Chernobyl employees not told shutdown button known to backfire. Or other known problems w/ rbmk reactor.
-titanic staff unaware ship sinkable, insufficient life boats. Even as sinking.
-NASA hierarchy never informed of O-ring weather concern.

What did “corporate-culture” look like?

-Perverse incentives “passed on”
-Information flow impeded

(I.e. lower level employees incentivized to cut costs, ignore risks. Not informed of known risks. Precluded from conveying red flags.)

E.g.

BP employees fired for raising concerns. Promoted for running shady safety checks, often w/o even visiting site.

NASA made clear to Morton-Thiokol contract up to renogiation. Time table essential. Showed frustratiin when engineers mentioned weather concerns.

My claim:

Bad corporate-culture+biased risk assessment.
<— consequence of perverse incentives.

Namely:

- Management has incentive to create this corporate-culture. And to generate these risk assessments.

- B/c helps ensures others act according to their interests. And can better justify, persuade them to do so.

Who cares?

1) understanding causality tells you what need to change to fix problem.

2) if miss causal structure, miss key insights on how beliefs, corporate culture work.

Namely:

If misidentify corporate-culture, or mistaken beliefs, as ultimate cause, tempted to adjust those w/o addressing source. W/ minimal effect.

If address perverse incentives, corporate-culture+mistaken beliefs will follow.

Illustrates common way beliefs influenced by incentives:

- beliefs biased in way that helps justify actions, persuade others to enact.
- e.g. perverse incentives that promote risk —> under-estimate risk.
-Motivated Resoning, restricted info flow <— means of achieving above.

Note:

-*very* different from beliefs biased by “what want to be true.” As social psych claims.

(NOONE wants disaster to occur. But that doesn’t explain why specifically those w/ perverse incentives to take risk ESPECIALLY prone to under-estimate.)

Note also:

-*very* different from standard econ approach of presuming beliefs determined by priors+information.

(Leaves NO ROOM for perverse incentives to come into play. Beliefs unaffected by what need to justify or persuade.)

Note:

Another key recurring feature of how incentives influence beliefs and behavior:

They often get “passed on.”

(Just like how climate denial helps big oil justify continued polluting. And they “pass on” these perverse beliefs to others they influence <—all republicans.)

Note:

Corporate-culture, like beliefs, don’t come from nowhere.

And don’t get far if just treat as exogenous. Or as resulting from some kind of inherent flaw in humanity or the mind.

Comes from incentive structure.

(Namely: Perverse incentive to take risk—> culture that promotes risky behaviors, and precludes info transmission re known risk factors.)

True of all “culture.”

Not exogenous force. Shaped by incentives.

(E.g. incentives of leaders? E.g. nationalism promoted by those who benefit from you sacrificing for their group? Like heads of states leading masses to slaughter in ww1, promoting nationalististic norms?)

EOM