Thread by @NoahHaber

Why "The Market for Lemons," a nearly 50 year old econ paper by George Akerlof, is nearly everything you need to know about why everything in science is the worst:

A thread brought to you by being late for the good train and being stuck on the local.

1/n

Gather 'round, and let me spin you a yarn about a tiny, elegant little paper that just sticks with you forever.

Think that one scene in A Beautiful Mind, but if that scene had correctly explained how the Prisoners Dilemma works instead of whatever the hell that was.

2/n

First thing: this is probably a very different kind of lemon party than what you might have been thinking.

Back in the ye olde pre-Twitter era of 1970, "lemon" was what you called a car that seemed fine on the lot, but was junk under the hood.

3/n

Disclaimer: the following is simplified/stretched/modified a little. Deal with it, econ PhD student class of 2023.

Alrighty, here we go!

There are two kinds of people out there: buyers and sellers. To get this analogy going, think scientists and science consumers/funders.

4/n

There are also two kinds of car for sale: lemons (🍋s) and peaches (🍑s, heh). It's really difficult to know if a car is a 🍋 or a 🍑 when it's on the lot.

In the original paper, buyers knew if the car was a 🍋, but we can relax that a little here.

5/n

Our version here needs only uncertainty about quality to work, not mal- or unethical intent.

Round 1: A bunch of cars go on sale at 🍑y prices, and a bunch of folks buy them. One such buyer is our friend Cave Johnson.

6/n

Lo and behold, some of those cars start breaking down. They aren't 🍑s at all! They are 🍋s!!!

And if you know ANYTHING about Cave Johnson, it is that you DO NOT GIVE THAT MAN LEMONS!

... and I have arrived at my stop, tale will continue momentarily.

7/n

... AND WE'RE BACK!

So, now everyone knows that, say, 10% of the cars are 🍋s, and expect that the market has 10% junk. This year, won't pay full 🍑 prices.

In the original, sellers can't afford to put peaches on the market at those prices, so they put more lemons up.

8/n

That starts a cycle: more 🍋s-> lower expectations -> lower prices -> more 🍋s, until eventually it's 🍋s all the way down.

And the analogy gets thin here, but that's just the strictest version of this story; we get roughly the same thing with a few modifications.

9/n

Firstly, remember it's really incredibly difficult to tell if a car is a 🍋or a 🍑. But there are some skin-deep signs, and there is a spectrum of really 🍑y looking 🍋s and vice versa.

Now, the "best" thing for a seller is to sell 🍋s that look like 🍑s on the surface.

10/n

In fact, if they DON'T do that, they'll just get out-sold by all the other sellers, and drop out of the market.

And remember: THIS DOES NOT REQUIRE MAL INTENT. It's just plain old competition over limited resources. Sellers who sell more true 🍑s will lose.

11/n

The kicker is that it's based on expectation: now everyone expects that all cars are lying 🍋s.

The classical way to "correct" this market and get it so that we have way more 🍑s and way fewer 🍋 is INFORMATION, or some way to tell if a car is really a 🍑or a 🍋.

12/n

Time to bring this back 'round to science.

We've got scientists (kinda like sellers) competing for limited resource $s from funders (kinda like buyers), and various other buyer-like folks who actually use the products of research, like doctors and such.

13/n

Making a research 🍑 is expensive, time-consuming, and riskier than making a research 🍋. But it's also incredibly difficult to tell them apart.

There are some really weak skin-deep and barely relevant, but easily visible signals, like p<0.05.

Sounding familiar?

14/n

So, the folks who produce 🍑y-looking 🍋s spend less to produce, and get more research $s, and outcompete the other folks.

And remember, those sellers don't have to be malicious; they may simply not know better, get lucky, or have slightly more lenient practices.

15/n

But wait, what about those expectations?

You mean like when people have been seeing the bajillionth egg/coffee/chocolate 🍋 over DECADES in a seemingly endless cycle of more and more BS?

Why are we surprised? We're a Market for Lemons!

16/n

As in the original, the (well, one) way to "fix" the market is to reveal the 🍋s and 🍑s for what they are. And that is INCREDIBLY difficult.

The reality is that our current peer review and publication systems are not even CLOSE to being able to get it done.

17/n

It will take enormous resources and painful reform to adapting or rebuilding our systems to focus on core quality, rather than skin-deep fuzz.

And, while there are some good signs for the future, it's unclear how much progress we're making here.

18/n

I'm trained as an economist, so this tale is a natural one for me. And it's maybe unsurprising that the tactic I and many others have taken to trying to fix things is to try to figure out how to sort 🍑s and🍋s.

But it's hard and slow progress, and frankly infuriating.

19/n

So every time I see one of those 🍋s in my news feed, I channel a little bit of my inner Cave Johnson, and will leave you with this.

Noah out.

20/20

Clarification: I don't mean that The Market for Lemons is like Prisoner's Dilemma, except in the sense that they are both toy models that work really nicely for a lot of situations well outside their original use.

FOOTNOTES TIME!

1) A correction: The "buyers" at the end of #5 should be "sellers" (credit to @jonatanpallesen)

2) Others have made this analogy, and far more eloquently
/ less snarky than I. E.g. this from @siminevazire (credit to @hardsci) collabra.org/article/10.152…

21/20

3) What do I mean by 🍋? Fraud is almost certainly not the biggest problem. Fraud is malicious, and this is not an argument about malice.

To me, the most serious, pervasive, and problematic 🍋 s, are weak and misleading "standard" methods (cont.)

22/20

3 cont) For example, it would be fair to call many (most? all?) of the articles we found to be overstated/weak in the CLAIMS study (metacausal.com/CLAIMS) 🍋 s, again noting no mal intent or fraud from the authors of those articles.

23/20

4) We may be in a market for 🍋 s, but it ain't all 🍋 s (looking at you antivaxxers/climate change deniers). More rigor often puts researchers at a competitive disadvantage holding all else equal, but not all else is or should be held equal.

24/20

5) If you are unfamiliar with Cave Johnson, quit whatever you are doing and play through Portal and Portal 2.

25/20