Thread by @bengoldacre

BING BONG NEW PAPER

@NDevito1 @sebbacon and me in the LANCET today.

We definitively assess compliance with US law FDAAA 2007, which requires clinical trials to report results within 12 months of completion.

The answer:

59% of trials breached the law

thelancet.com/journals/lance…

@NDevito1 @sebbacon This was a truly epic piece of work. We parsed the law. We converted it into unambiguous logic. We built software to download all US trial data and apply that logic. We stored snapshots for time trends. We built regression models to help spot patterns. Because of that we can say:

@NDevito1 @sebbacon Compliance with this law is very poor. But it is also *staying* poor: it is not improving over time. You can see for yourself in the blue line on this graph. That is very concerning. Why has there been no improvement? One key issue is likely to be enforcement...

... or rather, the absence of enforcement. @US_FDA is entitled to impose fines of up to $10,000 a day on non-compliant trial sponsors (that's now been inflation-adjusted to $12,103 a day). We calculate @US_FDA could have imposed $4bn in fines. They have never fined anyone, ever..

Of course we look at who is more, and less, likely to comply with the law on reporting trial results. Here's a funny thing. For many years the pharmaceutical industry poo-pooed efforts to get all trials reported. There was much funny business. But now that the law is passed?...

... drug companies are much better at complying than non-industry trialists. Why is that? It's hardly hard. Companies might lobby, energetically, against rules; they might hunt for loopholes, skillfully; but after that they also... competently follow those rules...

Competence is an interesting phenomenon to explore. Data scientists and data journalists might be interested in this next bit. Anyone can produce a paper or a finding with a vast budget. But research, and data analysis, is usually much more like a game of "scrapheap challenge"...

We have concocted a cunning new explanatory variable for these analyses to explore competence and experience among trialists. For every trial sponsor, we counted how many trials they have on the register in total. We split this into quarters. Now....

.. this means we now have four groups of trials: trials run by organisations that do LOADS of trials; trials done by small-scale chancers who have only done a few trials ever; dnd two groups in between. Magic! New insights! For free! And the answer?...

... Sponsors who do LOADS of trials are SPECTACULARLY more likely to comply with the rules on reporting results. (I think they're more likely to be competent in other ways too.. but that's for another paper).

We thought it would be helpful to give performance data for individual organisations. Here are the ten sponsors with the most trials on the register. Credit where it is due: we should praise the drug companies listed for complying so well with the law. By contrast...

It is remarkable to see @MassGeneralNews, Harvard Medical School, breaching FDA laws on clinical trial reporting so commonly: only 32/58 due trials from them complied with the law. @UCSF has the worse compliance: just 6/38 due clinical trials complied. thelancet.com/journals/lance…

@MassGeneralNews @UCSF So why does this all matter? Simple. Clinical trials are not abstract research projects. We use trials to produce the treatment guidelines used on all patients every day. Doctors and patients cannot make informed choices about which treatments to use when results are withheld.

Patients participate in clinical trials because we tell them (often explicitly in consent forms) that their participation will produce new evidence to improve treatment for all patients. When we fail to report results, we break that promise, and we breach patients' trust.

So, why should you trust our paper, and our findings? Because we did it right. We share our methods from beginning to end. We also share our entire codebase, to download and process the data, for review and re-use under open licenses. Go see: github.com/ebmdatalab/fda…

If you'd like to know more about why researchers should ALWAYS share their analytic code (and how often they fail, or even refuse to do so) see our recent @bmj_latest editorial on the topic.

https://twitter.com/bengoldacre/status/1197482152151175168

@bmj_latest Lastly, if you want to see whether your organisation has reported all its trials, and complied with the law, we share live updating data here fdaaa.trialstracker.net

We do this because we want to help those who want to comply, and change the incentives around papers like this. When you publish a one-off journalistic piece, or paper, then organisations who have failed to comply with the law are incentivised to tough it out...

...issue a statement "we take these matters very seriously...", maybe try to pick some minor holes or muddy the waters on some of the numbers, but overall, you're trying to survive the week until the critical numbers go away. We provide a better route for organisations: improve.

.. If any organisation is concerned by their poor performance in this data, or this paper, they can simply report their trials, and they will immediately rise to the top of the league tables. Our data updates EVERY working day...

.. and this isn't theoretical. Our trialstracker.net tools make it easy for good people to do the right thing, and here are examples of good people doing the right thing, using our data:

ebmdatalab.net/our-fdaaa-tria…

We assume good faith, and we are not disappointed.

So!

Please do read + share our paper, it's written to be accessible for all!
thelancet.com/journals/lance…

Sign alltrials.net

Check fdaaa.trialstracker.net

And huge thanks to @JohnArnoldFndtn for funding my transparency work, and @GoodThinkingSoc for funding the trackers!

Nice coverage by @reuters for our new Lancet paper out today, showing major US universities (and companies) failing to comply with @us_fda rules on reporting clinical trial results. And FDA, in turn, failing to enforce the law.

reuters.com/article/us-hea…

thelancet.com/journals/lance…