With LLMs for science out there (#Galactica) we need new ethics rules for scientific publication. Existing rules regarding plagiarism, fraud, and authorship need to be rethought for LLMs to safeguard public trust in science. Long thread about trust, peer review, & LLMs. (1/23)
I’ll state the obvious because it doesn’t seem to be obvious to everyone. Science depends on public trust. The public funds basic research and leaves scientists alone to decide what to study and how to study it. This is an amazing system that works. (2/23)
But it only works if scientists and the public each uphold their part of the deal. The pressure on scientists today to publish has never been greater. Publication and citation metrics are widely used for evaluation. (3/23)
In this environment some small number of scientists will cheat to increase the number of papers they write. Automated tools like #Galactica will assist them. Many have argued that warning notices on #Galactica are sufficient to prevent misuse. (4/23)
This ignores the fact that there are people who *want* to misuse it. They will see it as a shortcut to write more papers. Those papers will definitely not carry a warning that the text comes from an LLM. (5/23)
Some of these papers will slip through the review process. They will include incorrect or biased information. Reviews of the literature will be slightly off. Results will be fabricated. Other authors will be influenced by these papers. (6/23)
Science funding is not so plentiful that society, and scientists, can waste it pursuing dead ends based on fake papers. There will be press articles about paper mills using LLMs to create fake papers at scale. When this happens, public trust erodes. (7/23)
When public support declines, so does political support for funding. Society needs science today as much or more than it ever has. We have big problems to address. We can’t afford to squander public trust. (8/23)
Shouldn’t the review process catch fake articles? Are reviewers so easily duped? In fact, people are easily fooled by natural sounding text. Reviewers are already overloaded and they can't take on the task of rooting out a flood of LLM-generated papers. (9/23)
LLMs are not going away. Even though the #Galactica demo was taken down, the code remains on-line. Science reviewing and publishing is now in a battle against misuse of the machine. So here are some thoughts about peer review in the age of LLMs. (10/23)
It’s obvious that copying text from Wikipedia without reference is plagiarism. But it’s also relatively easy to detect with a web search. Now, what if an LLM is trained on Wikipedia and someone uses the trained model to generate text? Is it plagiarism? (11/23)
I think it is. If you argue that using LLMs isn't plagiarism, then this must be because the LLM created something novel and the author is not “copying” existing text. If another human writes something novel that goes in your paper, this person is considered an author. (12/23)
In this case, an LLM that generates text for a paper should be listed as an “author”. Authors, however, are responsible for the contents of the paper. That is, they are responsible for fraud and errors. (13/23)
If an LLM is an author, who takes responsibility if what it generates is wrong? Of course, authors have other responsibilities. If a paper’s results are challenged, the authors need to be able to explain how the results were obtained and produce evidence. (14/23)
Can LLMs live up to that responsibility? Can they explain themselves? Authors also need to disclose conflicts that might bias their work. What biases does an LLM have and can it disclose them? (15/23)
The above suggests that LLMs can't be “authors”. The only viable solution is to require citation of all text generated by LLMs using the same rules we apply to quoting text from any traditional source. The text goes in quotes and the source is cited. (16/23)
I’d be fine with this. It’s transparent. Of course, it is unlikely that anyone will do this. What people want is to have the computer write their paper and then pass it off as their own work. That’s scientific fraud. (17/23)
So, the other alternative is to ban the use of LLMs in scientific publications. Of course, this is unenforceable but that doesn’t mean we shouldn’t impose it. It gives people a warning and it provides a mechanism for punishment for detected violations. (18/23)
It may not sound like it, but I think research on LLMs is important. I use LLMs in my own research. The last thing I want to do is to slow down that research. So what can we do? What I call for is three things: (19/23)
(1) responsible dissemination of these tools that takes into account the risks, (2) change in the peer review process that addresses the risks, (3) research into “antidotes”. Today, only large companies can afford to train LLMs. (20/23)
They can also afford to train adversarial networks to detect fake science. If a company releases a science LLM, they should develop a companion network to differentiate its output from real science. They should make this network available to publishers for free. (21/23)
A stated goal of #Galactica was to help researchers "distinguish between the meaningful and consequential." Ok. Do that! Build the system that can distinguish between fake and real science. That would be useful. (22/23)
Acting now to introduce safeguards is necessary to protect the integrity of scientific publishing, prevent an undue burden on reviewers, limit fraud, and defend the public trust in science. (23/23)

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Michael Black

Michael Black Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Michael_J_Black

Nov 17
I asked #Galactica about some things I know about and I'm troubled. In all cases, it was wrong or biased but sounded right and authoritative. I think it's dangerous. Here are a few of my experiments and my analysis of my concerns. (1/9)
I entered "Estimating realistic 3D human avatars in clothing from a single image or video". In this case, it made up a fictitious paper and associated GitHub repo. The author is a real person (@AlbertPumarola) but the reference is bogus. (2/9)
Then I tried "Accurate estimation of body shape under clothing from an image". It produces an abstract that is plausible but refers to

Alldieck et al. "Accurate Estimation of Body Shape Under Clothing from a Single Image"

Which does not exist. (3/9)
Read 9 tweets
Nov 1
Who should be the last/senior author on a paper? How do you decide? What does being last entail? I get these questions a lot and it’s confusing because the last author is often a senior person, running a group & raising money. Do those things determine last authorship? No. (1/7)
The last author is ultimately responsible for the paper throughout the process including conception of the idea, writing, rebuttal, camera ready, talk, video, code, website, tweet, dataset, etc. They don’t do everything but make sure it all gets done. Like a conductor. (2/7)
They are responsible for the paper’s intellectual integrity. If there is a mistake or worse, it’s the last author who will take the blame. It is not to be taken lightly.The buck stops with them. If something goes wrong, they have to fix it. (3/7)
Read 7 tweets
Jul 29
Avatars are central to the success of the #metaverse and #metacommerce. We need different #avatars for different purposes: accurate #3D digital doubles for shopping, realistic looking for #telepresence, stylized for fun, all with faces & hands. @meshcapade makes this easy. (1/8)
For on-line shopping, clothing try-on, and fitness, an avatar should be realistic – your digital twin. You need a true digital double to see how clothing will look in motion. But, creating avatars that are accurate enough for shopping is hard. (2/8)
Since it’s hard to 3D scan everyone, digital doubles must be created from a few images or a video. Existing methods require users to wear tight clothes and have cumbersome capture protocols. @Meshcapade uses a single image of a person in any pose, making creation easy. (3/8)
Read 8 tweets
May 25
The 5 stages of rebuttal grief.
(1) Denial
The reviewers totally misunderstood my paper. The review process is broken. R1 was clearly a student who has never reviewed before. R2 doesn’t know what they are talking about. R3 hates me.
(2) Anger
I’m going to withdraw my paper. I’ll submit it somewhere else where other people will love it. I hate this conference and this field. The whole process is broken. Reviews are random.
(3) Bargaining
I’ll explain to the reviewers why they are so mistaken. I’ll convince them that my paper is great and that they are idiots. My reasoning will be so powerful, that they will be swayed and will accept my paper.
Read 6 tweets
Jul 15, 2021
There is a lot of good thought going into how to make @siggraph more attractive for authors of technical papers (e.g. from @AaronHertzmann). All good. But the differences between the physical @siggraph and @cvpr/@iccv/@eccv conferences also matter. (1/8)
1. Remember being a grad student? If you write a paper, your advisor sends you on a free trip. So cool. CVPR/ICCV/ECCV are in different and exciting places. SIGGRAPH is mostly in LA. Boring. Branch out! (2/8)
2. SIGGRAPH is huge but very little of it has anything to do with me. The technical papers session is tiny. A scientist is lost in the crowd. You bump into fewer people. You walk for miles. (3/8)
Read 8 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!


0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy


3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!