Thread by @ProfDFrancis

Confused by the Corona news on this wondrous day, after fending off weeks of intravenous laser bleach and chloroquisade te pointes?

Two (2) [TWO] trials reported on the same day.

Let's have a look!

2 Different trials on Remdesivir, an antiviral originally developed for Ebola, but having theoretical reasons to work in Covid.

Trial 1.

Chinese RCT in the Lancet today.
237 patients randomized. Unfortunately 2:1, which contributed to inadequate power. Also they ran out of patients as Covid settled down in Hubei.

Non-significant trend to speeding up recovery.

Trial 2.

NIAID trial, which issued a press release today.

(And Dr Fauci also did a chat from a sofa, for some reason. Probably didn't want to talk at podium lest the Dear Leader barge in and talk crap)

Placebo controlled, international.

Very convincing reduction in time-to-get-better.

Mortality trended towards benefit. In the context of a big reduction in time-to-get-better, I think we should give the mortality the benefit of the doubt.

So now I am a believer in remdesivir.

"But Dr Francis, you were laughing at it before, so you just reversed your opinion?"

Yes. It's called thinking.

I was laughing at bullshit stuff like Zinc, Zirconium, Vitamins A to Z, etc, and any drug dragged at random from the pharmacy.

When Remdesivir, like many drugs, just had useless data, I carried on laughing. What it needed was proper clinical trials.

It has had two now. It works

If I catch Covid:

1. Give me remdesivir.
and
2. Enrol me in an RCT of additional therapy too please, as there is a big chunk of death still to work on.

John Mandrola is right to ask for the paper so we can see the details.

However, this is an NIH trial conducted by proper experts in the field, at lightning speed, internationally.

So I start with the prejudice that the methodology will be tip-top perfect.

And I trust people with skills of Antony Fauci: he doesn't accept any old rubbish data.

But John linked to a useful thread of concerns. I will go through them one by one but here is a link to the original.

https://twitter.com/jopo899/status/1255591437460279297

Joanna is right to be skeptical of news stories in general because we have had an insufferable torrent of utter crap in these recent weeks, from vitamins, to intravenous bleach, to radiation burns of the trachea.

So let's have a look at her questions, which are very good.

There are different meanings of "preliminary", and within science we are so familiar with the meanings that we rarely explain them to non-scientists, and they rarely ask anyway.

"This is just the first 100 patients" might be a form of preliminary, where you are just reporting progress early in a study, at a scheduled "look-point".

This isn't that.

"This is just the first N days of followup. The trial will carry on for M years," is another form of followup.

This isn't that.

"We have given all the centres 10 days to send their results in, of who died or got better etc, but only half the centres have sent their answers by the deadline. So this is it, but it will change when the slackers get their arses in gear."

This isn't that.

"This is the primary endpoint result. We haven't written the rest of the paper yet."

This is that.

The only thing that would change the interpretation of this trial would be

(a) It wasn't randomized
(b) It wasn't blinded (with placebo control)
(c) Some other major lie

These have zero chance of occurring amongst established scientists reporting through the NIH

So I would accept this preliminary data, from this institution and scientific group, as solid gold.

Next question, equally good.

The trial was powered for the "getting better quickly" rather than "not dying" because binary endpoints (dead or not) convey only small amounts of information per patient, particularly when MOST PATIENTS DO NOT DIE.

It is LESS than one bit per patient. (h/t Frank Harrell)

They prespecified the "getting better quickly" as their primary endpoint, wisely.

I would not expect an underpowered mortality to show p<0.05. It happened to trend in the right direction, which if forced to guess I will take as consistent with a benefit (given what we know about the primary endpoint)

But my enthusiasm for the drug is from the primary.

It's only silly if

(a) people decided on it AFTER the data were available, or
(b) staff were unblinded, as they are in the pointless other remdesivir trials I was mocking.

But blinded, and prespecified, so kosher in my view.

Very good question here!

If it did exclude those who died, the trial would be mad.

This is where my prejudice plays its role. Whenever anyone with at least half a brain plans an endpoint of "time to get better", they would build into it a method of ensuring that people who die count very badly.

For example, setting an infinite time-to-get-better for them.

Or one of many cleverer methods that I don't know about.

My prejudice is that this trial was not cooked by one random dude like me but by a group of highly professional scientists who would have build this in.

Very good concern here, for non-randomized studies.

If we don't randomize, we spend endless hours, days and weeks afterwards arguing about which patients in the control group are equivalent at baseline to which patients in the active group.

And the result is heavily swung one way by the issue you raise.

But this is an RCT. In an RCT regardless of how sick you are, you are equally likely to be randomized to one arm as to the other.

The arms will not be identical in characteristics, but the statistics can reliably estimate the degree of wobble away from perfect balance.

Definitely Joanna.

One arm will always be sicker than the other. It is inevitable. However the extent to which that is the case is predictable, and that is the magic of randomization.

We can tell whether the difference in outcomes is greater than that which we could reasonably expect from the likely range of inequalities of sickness.

Steven Senn wrote an article about it that @drjohnm linked to recently.

Excellent point and quite right! No argument here.

Good question here. The answer is (c).

One weekend I will cover sample size calculations.

Ah but you won't need to do the tweetorial, I've just realised, after reading your next tweet.

8-)

Yes, they wanted to get an answer quicker, so picked an endpoint that would be more sensitive for improvement of the disease.

Good observation here.

Yes, I now take no notice of whether people describe their cohort as "moderate" or "severe" or whatever. It's less meaningful than we might first assume.

For example, one remdesivir trial I looked at on Twitter (not these 2 trials) defined severe covid as:

1. Needs to be in hospital
2. Needs Oxygen or has sats below some threshold

Trouble is, if you go in to hospital with breathlessness, you will get Oxygen. So you are severe!

So I do what you do Joanna, look at the control-arm mortality. That is an index of severity that is not subject to "inflation".

And your two other tweets are great points.

It's just that they are better directed at all the other rubbish stuff that has come out in recent weeks that is, effectively, of no value.

Or indeed negative value if uncritical people accept it as Gospel.

Some more great questions have emerged overnight!

This is really good. Like giving a talk to 1000 of the country's finest students/trainees, because they between them think up such a fine range of questions!

Yes Ivor, this is how clinical research really works!

I know you are surprised that I can be excited about the small relative and absolute risk reduction from statins, and remdesivir's effects which do not show significant survival advantage.

And unexcited about so much else.

"Scientists get excited when an effect is clearly non-zero,
Not when the point estimate of an effect is large."

D Francis
J Meta-statistical Musings 2020

A lot of people find this non-obvious, but the key is the error bar, namely the uncertainty in the result.

Suppose I do an RCT of 2 people. One gets Francovir, the other gets placebo.

The Francovir patient lives, the other one dies.

What is the *point estimate* of the effect size?

That is to say, with no other information, no background on the nature of the disease or of the patient, or of how easy it is to treat the disease (it might be as simple as dehydration, curable by any fluid)

That's the worthlessness of point estimates on their own.

The point estimate in my thought experiment is of a gigantic benefit. From 0% survival to 100% survival. That's an odds ratio for survival of roughly ... infinity!

Cue the world's shortest party.

But ...

What is the size of the error bar?

Exactly.

When you randomize just two patients, the two groups are as different as they can be.

One could be a 21 year old gymnast, who received 1 nanomol of intravenous chlorox

And the other a 108 year old diabetic with coronary artery disease who got placebo.

... in which case the survival of the one, and the death of the other, is nothing to do with the drug.

That's why we do lots of patients, so that a whole range of illness and wellness, amounts of viral load, and disease severity, go into each group.

The point of randomization is NOT to balance the arms for these.

It so happens that it does very roughly balance the arms for each variable, but that is not WHY we do it, I have realised recently, in this article by Stephen Senn. h/t @drjohnm
errorstatistics.com/2020/04/20/s-s…

We randomize, NOT TO BALANCE the arms, but to allow the statistical methods to CALCULATE THE LEVEL OF IMBALANCE that is bound to occur by chance.

This allows us to draw the error bar.

That is why scientists are more interested in the error bar, than the point estimate.

With a very small study, the point estimate is almost always wildly big (good or bad!) but the error bar immediately tells you that it is meaningless.

The p value is just shorthand for "The point estimate of the effect size is pretty far away from neutral, in comparison to the error bar, i.e. it is time to get excited."

Is the SIMPLE trial the one that forgot to have a control arm, and just compared two doses of Remdesivir?

I just ignored it.

It was a good thing it was unblinded. Saved a bit of money.

Could have saved more money by not doing it at all.

Oh, interesting!

So the so-called "last-ditch patients who would certainly die if we don't give them the magic stuff", actually were not!

There was no way to know because there was no randomized control group.

That is why the "compassionate use" terminology is pretty stupid here.

We should call it "compassionate killing".

Not because it kills the patient (although it might, if it was Francis Industries' iv Laser-Chlorox)

But because it kills everyone else who gets the disease afterwards, due to
(a) lack of good data, and worse
(b) adding of stupid data

No, this did not put me off. This is because I trust these proper scientists to have made the decision without access to the unblinded data, and independent of the company.

This is where reputation and track record matters: it affects what I assume when something is not stated.

Like if a surgeon does a CABG on me, and one week later, someone asks, "Do you you think she wore gloves? What makes you so sure?"

I would reply "Because she is an expert in surgery, working in a proper hospital surrounded by many professionals who would have freaked out if not"

Heh heh.

This is the situation of many people who are not so candid.

And this is why it is NOT patronising to censor stupid youtube videos on Covid, etc.

An entirely reasonable concern.

The reason I am letting it pass is that

1. I can understand what statistical method they would use for their NEW endpoint, but I sure as hell don't understand fully what they would have done for the OLD (seems rather complicated).

2. I have confidence that NIAID/NIH would not let people who have seen the unblinded data, anywhere near the endpoint modification process.

That is the nature of scientific reputation. It is based on decades of professionalism, which they have demonstrated repeatedly.

I have no doubt that the company are utter scallywags, Emo, based on all the dodgy doings you highlighted.

But I (currently) believe this trial and its interpretation were controlled by proper independent clinical researchers with high standards.

My advice?

1. Keep social distancing

2. Get remdesivir. It kills the virus a bit and will probably slightly reduce your chance of dying

3. Get randomized into further trials, so that you can help your loved ones to not die, rather than allow yourself to contribute to fake news