Thread by @Michael_J_Black

With LLMs for science out there (#Galactica) we need new ethics rules for scientific publication. Existing rules regarding plagiarism, fraud, and authorship need to be rethought for LLMs to safeguard public trust in science. Long thread about trust, peer review, & LLMs. (1/23)

I’ll state the obvious because it doesn’t seem to be obvious to everyone. Science depends on public trust. The public funds basic research and leaves scientists alone to decide what to study and how to study it. This is an amazing system that works. (2/23)

But it only works if scientists and the public each uphold their part of the deal. The pressure on scientists today to publish has never been greater. Publication and citation metrics are widely used for evaluation. (3/23)

In this environment some small number of scientists will cheat to increase the number of papers they write. Automated tools like #Galactica will assist them. Many have argued that warning notices on #Galactica are sufficient to prevent misuse. (4/23)

This ignores the fact that there are people who *want* to misuse it. They will see it as a shortcut to write more papers. Those papers will definitely not carry a warning that the text comes from an LLM. (5/23)

Some of these papers will slip through the review process. They will include incorrect or biased information. Reviews of the literature will be slightly off. Results will be fabricated. Other authors will be influenced by these papers. (6/23)

Science funding is not so plentiful that society, and scientists, can waste it pursuing dead ends based on fake papers. There will be press articles about paper mills using LLMs to create fake papers at scale. When this happens, public trust erodes. (7/23)

When public support declines, so does political support for funding. Society needs science today as much or more than it ever has. We have big problems to address. We can’t afford to squander public trust. (8/23)

Shouldn’t the review process catch fake articles? Are reviewers so easily duped? In fact, people are easily fooled by natural sounding text. Reviewers are already overloaded and they can't take on the task of rooting out a flood of LLM-generated papers. (9/23)

LLMs are not going away. Even though the #Galactica demo was taken down, the code remains on-line. Science reviewing and publishing is now in a battle against misuse of the machine. So here are some thoughts about peer review in the age of LLMs. (10/23)

It’s obvious that copying text from Wikipedia without reference is plagiarism. But it’s also relatively easy to detect with a web search. Now, what if an LLM is trained on Wikipedia and someone uses the trained model to generate text? Is it plagiarism? (11/23)

I think it is. If you argue that using LLMs isn't plagiarism, then this must be because the LLM created something novel and the author is not “copying” existing text. If another human writes something novel that goes in your paper, this person is considered an author. (12/23)

In this case, an LLM that generates text for a paper should be listed as an “author”. Authors, however, are responsible for the contents of the paper. That is, they are responsible for fraud and errors. (13/23)

If an LLM is an author, who takes responsibility if what it generates is wrong? Of course, authors have other responsibilities. If a paper’s results are challenged, the authors need to be able to explain how the results were obtained and produce evidence. (14/23)

Can LLMs live up to that responsibility? Can they explain themselves? Authors also need to disclose conflicts that might bias their work. What biases does an LLM have and can it disclose them? (15/23)

The above suggests that LLMs can't be “authors”. The only viable solution is to require citation of all text generated by LLMs using the same rules we apply to quoting text from any traditional source. The text goes in quotes and the source is cited. (16/23)

I’d be fine with this. It’s transparent. Of course, it is unlikely that anyone will do this. What people want is to have the computer write their paper and then pass it off as their own work. That’s scientific fraud. (17/23)

So, the other alternative is to ban the use of LLMs in scientific publications. Of course, this is unenforceable but that doesn’t mean we shouldn’t impose it. It gives people a warning and it provides a mechanism for punishment for detected violations. (18/23)

It may not sound like it, but I think research on LLMs is important. I use LLMs in my own research. The last thing I want to do is to slow down that research. So what can we do? What I call for is three things: (19/23)

(1) responsible dissemination of these tools that takes into account the risks, (2) change in the peer review process that addresses the risks, (3) research into “antidotes”. Today, only large companies can afford to train LLMs. (20/23)

They can also afford to train adversarial networks to detect fake science. If a company releases a science LLM, they should develop a companion network to differentiate its output from real science. They should make this network available to publishers for free. (21/23)

A stated goal of #Galactica was to help researchers "distinguish between the meaningful and consequential." Ok. Do that! Build the system that can distinguish between fake and real science. That would be useful. (22/23)

Acting now to introduce safeguards is necessary to protect the integrity of scientific publishing, prevent an undue burden on reviewers, limit fraud, and defend the public trust in science. (23/23)