Thread by @Moshe_Hoffman

A legitimate criticism imo of many laboratory experiments in behavioral lit.

(Lab experiments often measure immediate effects w/o considering how we will adapt.)

https://twitter.com/page_eco/status/1137004785939361792

One reason might be misleading:

Measuring effect of a proximate mechanism. When other mechanisms are available. And is a deeper driver.

For instance, suppose wanna know why people aren’t sensitive to number of sufferers when give.

You guess: people only imagine one representative sufferer. <—proximate mechanism.

You demonstrate this proximate mechanism by asking subjects to imagine ALL sufferers, and find their giving is NOW influenced by number of sufferers.

You conclude “representativeness heuristic” is what “causes” people to have “scope neglect.”

But...

What if *real* reason people are “scope insensitive” is b/c reputational benefits from giving are unaffected by number of sufferers?

Then maybe irl, if given a chance to adapt, even if asked to imagine all sufferers, they would find a workaround.

Eg they may strategically misinform themselves about the number of sufferers.

IMO this is a common problem in behavioral lit. A common reason lab experimental evidence of causality can be misleading.

Especially when showing the effect of a proximate mechanism. When many proximate mechanism are available. And is a deeper driver of behavior.

One way to tell this is a problem in your literature:

Is there ten proximate mechanisms that lead to same phenomena?

If so, the proximate mechanism probably isn’t the *cause* even if your lab experiment shows a causal effect.

(There are dozens of documented proximate “causes” of “scope insensitivity” in giving.)

Another way to tell this is a problem in your literature:

Ask self: if I remove this proximate mechanism, irl will people find a work around? If answer is yes, it’s probably not the “cause.”

A second reason lab experiments that don’t give subjects a chance to adapt can be misleading:

If lab context is not one our emotions, ideologies, intuitions etc designed to handle.

For instance, suppose measuring how much people give in one-shot anonymous dictator game. (Subjects asked to allocate $ between self and stranger. No reputational consequences.) <—not a task that naturally occurs irl

What does this behavior measure? Might think it measures our “genuine degree of altruism.” Or that it shows we give even when aren’t reputational benefits.

But this is misleading. Can see why by giving subjects a chance to adapt to this novel context.

Eg let subjects play similar games many times. Like twenty dictator games. Once each day. With a new partner each time.

Will they give less on the 20th day?

I bet.

Why?

Well they will feel less empathy toward stranger. Develop an excuse. No longer intuit this is an appropriate time to give. Etc.

They will adapt.

The original measure was misleading.

To conclude:

Lab experiments that don’t give subjects chance to adapt can be misleading.

Especially when showing proximate mechanism. And irl is deeper cause+exist workarounds.

Or when experiment uses novel context. And intuitions, emotions, etc will adapt, given experience.