David Smerdon Profile picture
Jan 27 20 tweets 5 min read Twitter logo Read on Twitter
Why does chatGPT make up fake academic papers?

By now, we know that the chatbot notoriously invents fake academic references. E.g. its answer to the most cited economics paper is completely made-up (see image).

But why? And how does it make them? A THREAD (1/n) 🧵 Image
ChatGPT is based on a language model, which assigns a probability distribution over sequences of words. A rough way to think about it: Given the start of a sentence, it will try to guess the most likely words to come next. (2/n)
A simplistic example: Give it “An apple a day…” and it will scan its immense library and come up with the most likely continuation: “…keeps the doctor away.” (3/n) Image
ChatGPT is of course more sophisticated than that, and it also ‘predicts’ the start of sentences, and ensures that whole documents are consistent. But the fundamental idea of predicting the next words in a sequence is still the same. (4/n)
Now consider the prompt “What is the most cited economics paper of all time”. The most ‘likely’ beginning to a language-based answer to this question is “The most cited economics paper of all time is”, which is what chatGPT spits out. (5/n) Image
No surprises there. But how does it choose the beginning of the title? It can’t scan papers themselves, but it can use website articles (inc. Wikipedia) that cite the titles of popular economics papers, and then use the words in the cited titles. (6/n)
Throughout the last 70 years, the two most common words in the titles of highly-cited economics papers have been “economic” and “theory”
(hat tip: Hugo M. Montesinos-Yufa and Brandon Brice, “The Era of Evidence”)
(7/n) Image
So we get the stem “A Theory of Economic”.
What comes next? The most probable word to finish this title consistently, given the pool of cited economics papers and the adjective ‘economic’, is “History”.
Now we have the title of our fake paper: “A Theory of Economic History”. We need the most probable author of this paper.
The most highly-cited author associated with economic history is Nobel laureate Douglass North.
Douglass North has been cited over 120,000 times according to Google Scholar, and his most cited work, the book Structure and change in economic history, bears similarity to chatGPT’s title.
But wait - how many authors should our fake paper have?
The most common number of authors in economics papers is 2.
We need another author, and someone who best fits a co-author to Douglass North on a paper called “A Theory of Economic History”.
Douglass North had many co-authors, but his most cited work with a co-author was “The rise of the western world: A new economic history”, with Robert Thomas.
(12/n) Image
So Robert Thomas is our co-author.
Finally, we need to choose a journal to publish our fake paper. Given the title and given the authors, which journal does chatGPT think is most likely?
Douglass North’s most-cited co-authored paper, "Constitutions and commitment: the evolution of institutions governing public choice in seventeenth-century England", was published in The Journal of Economic History in 1989.
But this isn’t the only reason!
When associating an economics journal to go with this paper, chatGPT also draws on ALL of the references to a journal that contain either “Douglass North”, “Robert Thomas”, or “Economic History”.
Douglass North became editor of The Journal of Economic History in 1960, and many website articles about the Nobel laureate reference this appointment. Combine this with his highly cited paper in 1989, and the choice is clear.
And why 1969 as the publishing year? Here, the AI’s choice is not as clear.
Douglass North published frequently in The Journal of Economic History from 1954-1992.
He was the journal’s editor until 1966.
The Nobel Committee cited his 1961 work as “Groundbreaking”, while his most cited work was in 1981.
Why exactly chatGPT chose 1969 is unclear, but to a human, it is as plausible a choice as any – as is the fake paper!
Postscript: An interesting and plausible hypothesis for why 1969 was chosen 👇

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with David Smerdon

David Smerdon Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @dsmerdon

Oct 5, 2022
My take on chess.com’s (CC’s) Hans Niemann report: a thread 🧵👇
The report is good. It didn’t need to be that long for the HN case, but on the other hand it’s great that CC has publicly laid out more details about its cheat detection system, the human process behind detection/banning policies, and their broader philosophy towards cheating. 1/
Unfortunately, the results of CC’s analysis are not what the chess world wanted to hear (not CC’s fault) because it means we are unlikely to ever get a resolution on whether HN cheated OTB. Super short summary: 2/
Read 9 tweets
Jun 6, 2020

What I like about this article:
- It corrects a commonly held misconception that Aboriginals in custody are more likely to die than non-Aboriginals (they're not)

What I don't like about this article:
- No reference to prison rates (ATSI are roughly 12x more likely to spend time in prison than non-ATSI. For women, it's 20x)
- Mentions child abuse and domestic violence in passing. But, no evidence that ATSI people have higher rates after controlling for other factors
What I'd like to see more of:
- Evidence-based discussion over the underlying causes of incarceration rates and what can be done
- Generally, more research around ATSI/non-ATSI inequality in income, education, access to health and affordable housing and their effects

Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!


0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy


3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!