Thread by @Ayjchan

Watch the UK Science and Technology Committee live sessions on reproducibility and research integrity:
parliamentlive.tv/Event/Index/7a…

At the moment, Dr Jessica Butler, University of Aberdeen and Dr Ben Goldacre, University of Oxford are discussing ways to update our 150-year-old scientific publishing system.

Butler says scientists are entirely judged by the number of publications in a small number of prominent journals. Your funding maps to your publications in top journals. So scientists are incentivized to please these top journals/editors. This may not lead to the best science.

Scientists compete for first authorship because only 1 or 2 scientists receive the credit for each paper.

Great comparison to surgery - you don't want your surgeons fighting for time during surgery.

You want to reward the team, not the individual.

Goldacre says we need professional software engineers in science today because of how much data is handled in research. It hasn't become mainstream because funders don't understand it - they're trapped in a model where everyone in academia is desperate to be an academic.

Funders don't understand that we are trying to seduce software engineers to help us in science.

💯 Need concrete measures, sustainable funding, acknowledgement of the essential value of full-time software engineers to research.

Butler says early career researchers are disincentivized to prioritize reproducibility. Majority of scientists are on short term contracts/grants. Just when you're done training, the funding runs out, visas end. The only thing they're judged on for the next step is their paper.

Discussion of how to force a culture shift, e.g., norms around data sharing.

Goldacre says funders can do a lot by mandating data deposition & open methods (e.g., GitHub for computational work). Reproducibility pipelines have shown promise.

Butler says simple pressure from above to make research open access, focus on quality/rigor not just originality, can aid research reproducibility.

Goldacre says grassroots efforts for reproducibility are not considered high value. Behaviors largely follow the funding.

Butler says her biggest worry about her public criticism of scientific publishing is that it could be weaponized to say science doesn't work. But it's important to speak frankly about shortcomings.

The 2nd set of expert witnesses, speaking on the role of publishers on research integrity:

Dr Ritu Dhand, Chief Scientific Officer, Springer Nature

Dr Elizabeth Moylan, Publisher, Research Integrity and Publishing Ethics, Wiley

Dhand said publishers have checklists for authors to check if they have shared their data/code/protocols. They support preprints and community norms but don't mandate it. Sometimes they police some of these norms.

The STC asks for evidence that journals are publishing both confirmatory and non-confirmatory research pieces.

Moylan says she will talk to her colleagues to see if they can share the evidence.

STC asks what incentives publishers have to publish reproducible work.

Moylan says they want to publish reproducible work. But did not cite any incentives.

I'm not envying the position Dhand and Moylan are put in right now.

Scientific publishing system is very, very outdated and suffers from many problems. There is no incentive/penalty for publishers with regards to research integrity.

STC says research integrity is of existential importance to the research industry. Publishers more likely to publish striking results than reproducibility studies.

Dhand said most studies are incremental. Specialized journals are interested in publishing these increments. No scientists are interested in reproducing work. Most science builds on published work.

Dhand points out that research funding bodies do not fund reproducibility studies. Incremental work verifies published results.

My question: Who is tracking if a top paper has been independently built upon?

STC asks what process is in place to facilitate constructive criticism of published work? To make sure we're not building more research on top of unsound work?

Dhand said scientists who can't reproduce a published experiment can write to journals.

STC asked do you publish those letters?

Dhand said Nature publishes these criticisms as Matters Arising.

Speaking from personal experience of trying to publish our constructive criticism of the Xiao et al. @nature and Liu et al. @PLOSPathogens #PangolinPapers that were not reproducible, I'll say these criticism letters are not commonly published.

https://twitter.com/Ayjchan/status/1459206946184122378

@Nature @PLOSPathogens Dhand says the scientific community will start talking about a non-reproducible paper, so it cannot be hidden.

STC says these corrections take a long time.

Dhand says it's because these are like investigations so they must take a long time.

STC asks if there is appetite among publishers to create systems to pick out rare cases of misconduct.

Moylan said absolutely but can take years to figure out each case. Dhand concurred, said PI is responsible for assessing the raw data, but not all PIs have time.

Dhand said there's no position in labs where the person assesses the quality of data for publications...

I have to say this session is very difficult to watch.

Final session on #OriginOfCovid

STC questions Richard Horton, Editor in Chief, The Lancet about @TheLancet letter organized by Peter Daszak of @EcoHealthNYC to condemn "conspiracy theories suggesting that COVID-19 does not have a natural origin".
thelancet.com/journals/lance…

Horton says purpose of letter was to say, instead of blaming Chinese scientists, we should be offering solidarity. And to say, be careful about raising speculations because we won't have evidence one way or the other about the #OriginOfCovid

STC asks how much attention was paid to COIs?

Horton said authors are asked to declare COIs. @TheLancet takes those statements on trust. There were indeed significant COIs particularly in relation to Peter Daszak.

My question: what's the penalty?

Horton said it took over a year to persuade Daszak that his affiliation was a COI.

STC asks if Horton has considered any changes in the publication process to guard against this happening again.

Horton said they're publishing 100s of authors every week, they don't investigate every author, they take it on trust.

There's now heightened awareness around this issue so Horton says they're more careful about who they publish. But there has not been any change in their publication process.

STC asked if Horton acknowledges that the letter closed down discussion around a possible lab #OriginOfCovid

Horton said he is aware that the silencing issue has been raised.

Horton said @WHO has not been silenced during their joint study with China on the #OriginOfCovid

STC said the WHO team was denied access to data.

Horton said the WHO team was able to put a lab #OriginOfCovid on the table.

STC asked if Horton did not think a lab origin worthy of consideration when he published the Feb 2020 Lancet letter.

Horton said he agrees with WHO view that lab origin is extremely unlikely.

I reminded the room about the unscientific process that the China-WHO team used to evaluate #OriginOfCovid hypotheses.

Step 1. Ask the suspects if they audited themselves.

Step 2. Everyone votes in the presence of Chinese government minders.

https://twitter.com/Ayjchan/status/1359892457597652994

FYI Peter Daszak had been invited to be a witness in this last session but he declined. Otherwise I would have put questions to him about the rigor of the @WHO joint study with China and why he has been and still is sitting on so much critical #OriginOfCovid information.

One point that I pushed for aggressively, in the theme of research integrity, is the release of all original covid/bat CoV manuscripts submitted to journals in the early days of the outbreak (and even before). We know many of these were withdrawn/rejected/altered.

It's vitally important that independent scientists can review these submissions to understand if there is more Covid-related data that has been obscured. And to identify (& fix) loopholes that can be exploited in our scientific journals, which have a cost measured in human lives.

The expert witnesses from scientific journals seemed to not have many ideas on how they can help to make research integrity a priority.

I'm very glad to respond to the STC's invitation to submit detailed recommendations on how publishers can incentivize research reproducibility.

The recording of the @CommonsSTC event can be watched here:

https://twitter.com/CommonsSTC/status/1471092412252622849?s=20