Thread by @CMichaelGibson

How can there be disagreement between groups about the effectiveness of a drug in an interim analysis? I have NO specific information about the AZ analysis, but let me make some GENERAL comments (thread)

The timing of an interim analysis is often pre-specified (planned in advance) so that the date cannot be “cherry picked” to “make the results look good”. In this case it was pre-specified that events that occurred through February 17th would count.

Can you imagine the criticism researchers would face if they changed the date? Could you imagine the criticism if the results were reversed such that through Feb 17 there was 76% effectiveness & AZ decided to report 79% based on “more recent”preliminary data?

To avoid “cherry picking” or “moving the goal posts” we pre-specify either the date or the number of events that will be included in the interim analysis.

What has to happen before such an interim analysis? An independent group, called a clinical event committee (CEC) has to review each case reported by the sites to make an independent judgement as to whether a patient developed Covid or died from Covid in this case.

There is not infrequent disagreement between the doctor caring for the patient and the independent CEC group as to whether the patient had an event or not (in this case developed Covid or not or died of Covid or not).

The independent CEC uses very specific criteria to decide if an event occurred and this is pre-specified in a document called the CEC charter. The CEC is provided access to the patient’s records (the raw source documents) to make this determination. They don’t know who got drug.

When it comes time to do the interim analysis sometimes not all the events have been independently reviewed by the CEC because there may be for instance additional documents that were requested by the CEC. In this scenario the sites determination is used for “best available” data

At the end of the study ALL THE EVENTS will have been reviewed by this independent CEC committee and there will be no use of site reported events.

Why can’t you just use “the most up to date events”? The events may be determined by the independent CEC committee to NOT have occurred in say about 5% to 25% of those recent cases depending upon the type of event that is being looked at.

Clearly preliminary reports of death are clear cut and should count, right?

No.

The independent CEC committee may disagree with the site as to whether someone died of heart disease or in this case Covid.

Usually the DSMB and the researchers (or sponsor) have INDEPENDENT STATISTICIANS. Can these 2 groups of statisticians disagree? After all they are working from the same pre-specified statistical analysis plan and the same data?

YES THEY CAN!

Usually there is one independent statistician doing the analysis for the independent data safety monitoring board. One.

The researchers or sponsor generally have two or more statisticians working independently of each other to verify the results.

Usually the 2 statisticians are given fake/scrambled or dummy data to see if they get the same results with scrambled data. If they don’t there are meetings to reconcile the discrepancy in the results. Once they agree, then the REAL treatment assignment is inserted in the data.

We usually begin having 2 statisticians undertake the dummy data analysis 3 to 6 months before the trial is over.

3-6 months.

Yes it can take a long time to reach consensus between statisticians when thousand of patients are studied.

It was therefore understandable when the sponsor indicated that it was going to take 48 hours to analyze the data. There are numerous checks and balances in place to assure the accuracy of the analysis. Likely 2 or more statisticians had to concur on the results.

So how can the one statistician from the DSMB come up with a different result than the consensus of multiple statisticians at the company?

1. Use of different datasets to work from.

2. Different interpretations of the statistical analysis plan.

Not sure what happened here.

The FDA will conduct its own independent analysis of the data which might yield a third result.

Nonetheless, at 76-79% effectiveness and no deaths or serious illness this vaccine appears to be an additional tool in our battle with Covid.

Stay tuned!

People are asking "why didn't the company just report the DSMB numbers".

Likely because there are rigorous protocols in place that AZ must conduct the analysis which is verified independently by 2 statisticians rather than taking the word of the lone DSMB statistician.

In 34 years of clinical trials I have never seen an interim analysis reported based on what a DSMB statistician reported. It is always the researchers / sponsor analysis that is reported with 2 independent statisticians verifying the results.

Could you imagine the outcry if the DSMB analysis was more favorable, and AZ chose to report that rather than a less favorable number in their own analysis?

All these procedures are pre-specified so no one can choose the data that they wish to report.

The sponsor pre-specified a cut off date for events through February 17th for the analysis.

They are compelled to report the result of a pre-specified analysis (not a more or less favorable one) because the pre-specified analysis is "material" to a publicly traded company.

Reporting of pre-specified analyses minimizes the potential for bias and is a mainstay of good statistical practices.

The sponsor appears to have followed this process.