Thread by @nabarund

I'm seeing epidemiologists make a logical fallacy about the COVID vax + blood clots. Saying the risk (or rate) is "1 in a million" = misleading
Here’s a quick breakdown on how to do better by #pharmacovigilance (PV) stats
Audience: #epitwitter #datascience #RxEpi

2/ Comparing to birth control risks isn't proper. The *type* of clot is different. But also, quantified risk of clots from The Pill are from studies where each patient was assessed for the outcome (side effect). That's not so with COVID vaccine data now

https://twitter.com/nabarund/status/1382006242680438786?s=20

3/ In some of the COVID vax clinical trials, only 1-out-5 had systematic detailed assessment of side effects. For the other 80%, the trials relied on "spontaneous reporting"

https://twitter.com/nabarund/status/1337105506540924931?s=20

4/ The first 6 cases were found through “spontaneous reporting” - cases were reported when a patient or provider felt it was warranted to report it. Not every patient was assessed or reported. These data collection systems are maintained by pharma and are legal requirements

5/ Patients and HCP are encouraged to submit reports even if they aren't totally sure if the side effect is related to the drug or vax. The intent is to cast a wide net. This works generally well to *generate hypotheses* for rare, serious reactions. Not meant to be an epi rate!!

6/ Primary sources include: calls to companies, conversations with sales reps, scientific reports, clinical trials, news media, and patient support programs... A heterogeneous mix requiring human review. Sometimes conclusions are limited because reports can are thin on detail.

7/ In essence these are numerator-only data. In epi terms, there is strong potential for misclassification bias. This arises from incomplete case finding. In the strict epidemiologic sense, these numbers should not be interpreted as a risk nor as a rate.

8/ The numerator (side effect cases) does not arise exclusively from the denominator (doses administered). For example, RCT in other countries are included in the numerator, but not usually the denominator. Accidental ingestion by an infant of a dropped pill is another example

9/ We are now in a phase of "stimulated case reporting" - More reports are being found because there is more attention to the association. This is a well known phenomenon, and further warns against calculating an epidemologic risk from these data
ncbi.nlm.nih.gov/pmc/articles/P…

10/ As expected in stimulated reporting, 2 additional cases were found last week, including 1 from RCT. This case happened earlier than the 6 index cases, but was found after. Ergo, reporting date and event date should be analyzed separately.
nytimes.com/2021/04/13/hea…

11/ Clinical trial cases are often particularly helpful because of extensive baseline standardized biometric measures collected in the RCT. In practice, PV data are useful for generating new hypotheses with widespread use. But causality is central.

who.int/medicines/area…

12/ CAUSALITY is assessed separately twice. First, each report is assessed on an individual case basis taking into completeness of the record,  time-to-onset between taking the med/vax and start of side effect, symptom cluster, reporter qualifications, and general plausibility.

13/ The second causality assessment is a multi-disciplinary synthesis, looking across similar cases: animal data, clinical trial reports, sci lit, pharmacology, and background prevalence of the side effect in untreated people. This is our current phase.
ema.europa.eu/en/committees/…

14/ The need for background rate assessment was anticipated. Here is the list of some FDA was tracking proactively. These data are from Feb 26, and include only Pfizer and Moderna COVID vax. This analysis is being repeated this week for blood clots.

15/ Since COVID can cause blood clots, and there are lots of types of clots, there are valid questions about misclassification, coding schema, and detectability. Basic epi concepts, but the data are not collected from a single study! Oy vey, it's complicated

16/ But it is still useful to have a sense of how much drug/vax has been administered. First we must acknowledge that this is a qualitative assessment masquerading as a number.

17/ It is tempting to calculate a rate by putting the number of cases in the numerator and dividing by the number of doses administered. But this is misleading because not everyone was assessed for the side effect. Like, cases could arise outside the country (not in denominator)

18/ In scientific papers, when pharma authors do this to show how safe their products are, I always flag it and ask for a change. It's a false sense of security. And it's a "rate" that can only go up with time, as I've flagged before. Don't fall into trap!
ajph.aphapublications.org/doi/full/10.21…

19/ Best practice is to show a timeline of cumulative doses on one graph, and number of events on another graph separately. In text, write separate sentences with the number of doses and number of events. Putting them in the same fraction/graph/sentence is misleading.

20/ General solution to numerator-only data is the Proportional Reporting Ratio (PRR). For Vaccine (X) and a specific Side Effect (Y), the ratio of cases is compared to all other side effects for all other products in the database. Clearly, selection of comparators is critical.

21/ Bayesian methods are also used, and are particularly helpful because they stabilize over time.
openvigil.sourceforge.net/doc/DPA.pdf

22/ But more cases are needed to be able to use these stats. And the stats are meant of "signal detection" -- meaning to generate a hypothesis for multi-disciplinary review. We already have that with the vax-clots in question

23/ So, resist the temptation to use rates or fractions to contextualize the risk. These 6-8 cases could end up having different etiology. And more cases could be identified, and then found to not be related. This is a sensitive week where nuance matters.

24/ We should also respect that it is completely rational to wait for more info on side effects before committing to a vaccine. Steamroller vaccine encouragement based on faulty numbers does nobody any good. Hope this gives you fodder to explain what's actually happening.

/end

@threadreaderapp unroll