Thread by @RyanReedHill

Hey academics on Twitter:

Have you ever been scooped?
Do you worry about getting scooped?

My job market paper estimates the consequences of getting scooped in science using cool data from structural biology.
Let’s do a thread!

The paper is called:
Scooped! Estimating Rewards for Priority in Science

It’s co-authored with my awesome classmate @carolyn_sms , who is a fantastic collaborator. Look for her on the econ market next year!

Paper link: economics.mit.edu/files/18001

Economists have long characterized the reward system for innovation (patents, academic papers, etc.) as winner-take-all races. This extreme allocation of credit affects how we think about R&D investment, innovation strategy, and the pace and direction of science.

But there is almost no empirical evidence about the actual allocation of rewards in these races.
What is the effect of getting scooped?
In a close race, how much more credit do we give the “priority” (ie first) publication in terms of journal placement, citations, etc.?

Estimating the cost of getting scooped is tricky!
The key data problem is selective abandonment. If I get scooped, I may decide to drop the project altogether, or change the paper such that it no longer resembles the original paper.

We tackle this challenge in a very cool setting called Structural Biology. These folks figure out the shape of proteins in our cells – not a trivial task! They grow proteins into crystals, shoot them with x-rays, then make a model of the structure based on the resulting data.

Before anyone can publish a structure discovery, they are required to “deposit” their findings into the Protein Data Bank (@rcsbPDB ) confidentially. This worldwide repository covers the universe of structure discoveries, and it is the key to solving our empirical problem.

Here’s what we do:
1) Link identical projects by different teams using the amino acid sequence that defines the proteins
2) Find cases where the two teams independently deposited around the same time
3) Compare outcomes of winning and scooped projects

These are timelines from our slides that illustrate. We say project A scoops project B if:
1) Team A releases before team B
2) Both teams had deposited before Team A releases

Here is an example, including the actual structure image produced independently by both teams. Our regression sample is 1,630 races that look something like this.

So what do we find? Scooped projects are 2.5% less likely to be published, are 18% less likely to appear in a top journal, and receive 28% fewer citations.

Bummer! But are these effects “big” or “small”?

Another way to describe the citation result is that scooped papers get 42% of the total citations to both projects. That’s way bigger than the 0% that we usually assume in theoretical models of racing. It’s not winner-take-all, but maybe winner-take-much.

Since the fear of getting scooped is so prevalent, we compare our findings to scientist perceptions. We surveyed 915 structural biologists and posed a hypothetical priority race. Turns out scientists are very pessimistic about the probability and cost of getting scooped…

Respondents think there is a 27% chance of getting scooped between project completion and publication, much higher than the 3-8% chance in our data. They guess that they will only get 26% of the total citations if scooped, much lower than the 42% we find.

Some other interesting results from the paper: Scooped papers are less likely to be downloaded by readers, featured in news articles, cited in Wikipedia, and – you guessed it – mentioned on Twitter dot com.
(Data from @altmetric )

The effect of getting scooped varies by pre-existing reputation. This plot shows the citation shares in “mismatched” races between high- and low-reputation teams. Getting scooped has bigger effects if you’re from a low-ranked university or have very little publishing experience.

Lastly, we look at the effect on careers, and find that in this setting the costs are small, even for novice scientists. Getting scooped lowers citations to future work, but has minimal, if any, effect on publication rates over then next 5 years.

So yeah, nobody likes getting scooped. It sucks!

Here’s the good news: The evidence we present suggests that scooped projects are still very likely to get published and cited. Also, getting scooped is not a death knell for a fledgling career. Yay!

But maybe some bad news?

Our next paper considers the efficiency and welfare of scientific racing. We look at objective quality metrics in the PDB and present evidence that racing is associated with lower quality science. Tradeoffs will be discussed!

Thanks for reading, please check out the paper, and stay tuned for more research about research.

Website link: economics.mit.edu/grad/ryanhill