Thread by @NickAnderegg

This isn't a major paper, but it's an interesting jumping-off point for three different topics:

- Accuracy of RATs—in practice
- Understanding what descriptive (incl. Bayesian) statistics mean
- HOW rapid tests work

Here's a thread written for a general audience!

1/

This study was conducted from January 2020 to June 2021 using admission screening swabs from 556 oncology patients at a single hospital in Jerusalem.

The patients in this study were swabbed for both PCR and RAT, allowing for comparison of the detection ability.

2/

The takeaway is simple: The Rapid Antigen Test (RAT) used here had a sensitivity of 69.6%.

Sensitivity is the *true positive* rate. This means that, out of the patients who tested positive for SARS-CoV-2 using qRT-PCR testing, only 69.6% were *also* positive on the RAT.

3/

Additionally, specificity (true negative rate) of the RAT is 100%, which means that 100% of the patients who were negative on the qRT-PCR were also negative on the RAT.

However, we can also consider these values from a totally different (probabalistic) perspective...

4/

Positive and negative predictive values (PPV & NPV) reflect how well a test predicts a condition.

PPV is, essentially, the probability a positive TEST result predicts an actual positive COVID case.

Here, positive RATs had a 100% chance of accurately predicting a COVID case.

5/

NPV is, conversely, the probability a negative TEST result predicts an actual negative COVID status.

In this study, negative RAT only had a 92.9% chance of accurately predicting a negative final diagnosis for COVID.

So why four different numbers? What the hell do they mean?

6/

It'll actually be easier to explain the statistics if we derive them from scratch! These stats are calculated with simple arithmetic!

So, I started by loading the data into a set of descriptively-named variables we can use for the calculation:

7/

Another way to think of sensitivity is that it's the TRUE POSITIVES detected by RATs, as a fraction of the TOTAL PCR POSITIVES, which is the "gold standard" test, in this case.

(Specificity is more relevant than here if/when there is higher risk of false positives.)

8/

PPV and NPV differ from the above in that they're derived from Bayes' theorem, and they factor the baseline positivity rate of the tested sample into the calculation.

In this study, the prevalence of COVID among the tested sample was 20.1%.

9/

We can use the sensitivity, specificity, and prevalence values we calculated above to derive the PPV and NPV.

THIS is why the accuracy of diagnostic tests decreases as the population-level positivity rate increases: Significant interaction between prevalence and accuracy!

10/

False Omission Rate is the inverse of Negative Predictive Value. This means the probability of a COVID case being missed by a RAT—at the population level—is 7.1% *when you factor in prevalence*!

The probability of tests on different days both missing a COVID case is 0.5%.

11/

Why is prevalence part of the calculation? Let's see how it impacts the outcome.

Here are the PPV and NPV calculations for tests for hypothetical conditions which affect:

1. 100% of the population
2. 80% of the population
3. 50% of the population
4. 0% of the population

12/

And now you can see why you need to have two negatives, two days in a row on rapid antigen tests to consider it a true negative: variable likelihood of *what* is causing your symptoms, *when* you were infected relative to today, etc., means false negatives vary!

13/

Anyway, back to the paper! In qRT-PCR testing, the Ct value is a "relative measure of the concentration of target in the PCR reaction."

That is, it's an arbitrary value that is *consistently meaningful* for all machines running this specific test.

14/

This study found that if the qRT-PCR threshold was set to a value of 20—indicating the positivity threshold was crossed after 20 or fewer amplification cycles—the sensitivity of RATs was 91.8%.

RAT sensitivity dropped to 77.5% for those with PCR positivity between 20-30 Ct!

15/

What does it mean? Well, a lower qRT-PCR Ct value corresponds to *higher* viral load, so in an immunocompromised group (oncology patients), RATs are *somewhat* reliable at detecting COVID cases.

In this patient group, viral load skewed higher (indicated by lower Ct values).

16/

The biggest caveat to this study is that they didn't have symptom info for all cases. This can had an impact on the effectiveness of RATs: "Most studies agree on the fact that RAT can be mostly reliable in patients with respiratory symptoms and not asymptomatic individuals"

17/

What's the takeaway? If we're following the precautionary principle:

- RATs shouldn't be relied upon for *ruling out* infections.
- RATs still CAN be used to quickly and effective *rule in* an infection.

18/

Note that the data in the paper is pre-Omicron. On top of that, the RAT used here requires a nasopharyngeal swab to be taken by a professional.

All that is to say that the numbers here should probably be taken as the UPPER BOUNDARY for RAT reliability in the Omicron era.

19/

IMO, the reliability of RATs is probably much lower today, because:

- self-tests already have a lower reliability, and
- other studies have shown that Omicron seems to produce lower levels of antigen presentation.

Both of those could increase the false negative rate.

20/

Why is there such a big difference between RAT and PCR sensitivity? They work in fundamentally different ways!

qRT-PCR detects the presence of genes which encode: 1) the enzyme the virus uses to replicate, 2) the nucleocapsid gene, 3) the envelope gene.

21/

For this rapid antigen test, in contrast, the test line is coated with an anti-SARS-CoV-2 antibody, which reacts with a SARS-CoV-2 antigen.

The control line is coated with an anti-chicken IgY antibody, and the buffer solution contains a chicken IgY protein to react with.

22/

So while RNA from SARS-CoV-2 genes is *amplified* in qRT-PCR testing to allow even small amounts of RNA to be detected, rapid antigen tests just have to work with whatever is on the swab. If the right antigen isn't at the swab location, the test will be negative!

23/

If we were still in a world that practiced basic infection control, this study would have confirmed that rapid antigen tests are an effective measuring for rapidly detecting infections, to minimize exposure as much as possible.

24/

This thread ended up being more of a statistics lesson than anything, which I'll definitely be linking to a whole bunch.

The paper was published on August 2, 2024 in PLOS ONE, and is available open access:

25/25journals.plos.org/plosone/articl…