Thread by @dsmerdon

Why does chatGPT make up fake academic papers?

By now, we know that the chatbot notoriously invents fake academic references. E.g. its answer to the most cited economics paper is completely made-up (see image).

But why? And how does it make them? A THREAD (1/n) 🧵

ChatGPT is based on a language model, which assigns a probability distribution over sequences of words. A rough way to think about it: Given the start of a sentence, it will try to guess the most likely words to come next. (2/n)

A simplistic example: Give it “An apple a day…” and it will scan its immense library and come up with the most likely continuation: “…keeps the doctor away.” (3/n)

ChatGPT is of course more sophisticated than that, and it also ‘predicts’ the start of sentences, and ensures that whole documents are consistent. But the fundamental idea of predicting the next words in a sequence is still the same. (4/n)

Now consider the prompt “What is the most cited economics paper of all time”. The most ‘likely’ beginning to a language-based answer to this question is “The most cited economics paper of all time is”, which is what chatGPT spits out. (5/n)

No surprises there. But how does it choose the beginning of the title? It can’t scan papers themselves, but it can use website articles (inc. Wikipedia) that cite the titles of popular economics papers, and then use the words in the cited titles. (6/n)

Throughout the last 70 years, the two most common words in the titles of highly-cited economics papers have been “economic” and “theory”
(hat tip: Hugo M. Montesinos-Yufa and Brandon Brice, “The Era of Evidence”)
(7/n)

So we get the stem “A Theory of Economic”.
What comes next? The most probable word to finish this title consistently, given the pool of cited economics papers and the adjective ‘economic’, is “History”.
(8/n)

Now we have the title of our fake paper: “A Theory of Economic History”. We need the most probable author of this paper.
The most highly-cited author associated with economic history is Nobel laureate Douglass North.
en.wikipedia.org/wiki/Douglass_…
(9/n)

Douglass North has been cited over 120,000 times according to Google Scholar, and his most cited work, the book Structure and change in economic history, bears similarity to chatGPT’s title.
scholar.google.com/citations?hl=e…
(10/n)

But wait - how many authors should our fake paper have?
The most common number of authors in economics papers is 2.
We need another author, and someone who best fits a co-author to Douglass North on a paper called “A Theory of Economic History”.
(11/n)

Douglass North had many co-authors, but his most cited work with a co-author was “The rise of the western world: A new economic history”, with Robert Thomas.
(12/n)

So Robert Thomas is our co-author.
Finally, we need to choose a journal to publish our fake paper. Given the title and given the authors, which journal does chatGPT think is most likely?
(13/n)

Douglass North’s most-cited co-authored paper, "Constitutions and commitment: the evolution of institutions governing public choice in seventeenth-century England", was published in The Journal of Economic History in 1989.
doi.org/10.1017/S00220…
(14/n)

But this isn’t the only reason!
When associating an economics journal to go with this paper, chatGPT also draws on ALL of the references to a journal that contain either “Douglass North”, “Robert Thomas”, or “Economic History”.
(15/n)

Douglass North became editor of The Journal of Economic History in 1960, and many website articles about the Nobel laureate reference this appointment. Combine this with his highly cited paper in 1989, and the choice is clear.
(16/n)

And why 1969 as the publishing year? Here, the AI’s choice is not as clear.
(17/n)

Douglass North published frequently in The Journal of Economic History from 1954-1992.
He was the journal’s editor until 1966.
The Nobel Committee cited his 1961 work as “Groundbreaking”, while his most cited work was in 1981.
(18/n)

Why exactly chatGPT chose 1969 is unclear, but to a human, it is as plausible a choice as any – as is the fake paper!
(19/19)

Postscript: An interesting and plausible hypothesis for why 1969 was chosen 👇

https://twitter.com/arekfurt/status/1618966238050525184?s=20&t=hilHKlcGCAWg0Sookm0kzw