Thread by @houndcl

Having seen so many double standards on #remdesivir and #hydroxychloroquine result, I decide to start a thread to share how I read a scientific paper.

While I only expect 1% people who finally adopt the protocol, I do hope scientists and famous writers have a try.

thread🔻

(1/) Links:
Remdesivir (NEJM)
nejm.org/doi/full/10.10…

Hydroxychloroquine + azithromycin (preprint)
mediterranee-infection.com/pre-prints-ihu/

(2/) I will use the NEJM paper as an example,

As most people will do, my eyes were caught by the title & conclusion first.

(3/) The 2nd step may be distinct from general reader.

Based on title & conclusion, I will define the CONTEXT and EVIDENCE CHAIN that paper needs to present to reach such conclusion, before I jump to read method and result.

(4/) CONTEXT defines the scope of conclusion that a study should and could reach. This is important because not all the studies are the same. e.g., this paper’s context is

-Compassionate use of Remdesivir

(5/) The context “Compassionate use” defines a study

-without control group

-unable to reach conclusion regarding the general efficacy/safety related to a disease

-with strong patient selection

(6/) Therefore, scope of conclusion is quite limited, e.g.,

“drug X improved (or worsened) ABC outcome in N patients”

“drug X is generally safe in N patients”

“strong side effect was observed from N patients treated with drug X”

(7/) any conclusion below is NOT what we expect from a compassionate use study.

“drug X improves ABC outcome”

“drug X is safe / shit”

“drug X leads to strong side effect”

(8/) In comparison, when an RCT was conducted, we DO expect the above conclusions.

If an RCT paper only concluded “drug X improved (or worsened) ABC outcome in N patients”, it means the RCT doesn’t work.

Again, CONTEXT defines the what conclusion should and could reach.

(9/) to reach the conclusion “… patients hospitalized for severe Covid-19…, clinical improvement was observed in 36 of 53 patients (68%)”, we need a chain of EVIDENCE. This involves

definition of terms &
supporting data

(10/) for this NEJM paper, the following EVIDENCE must be presented:

how to define clinical outcome(s)

how these outcomes should be combined to define “improvement”

how to define “severe” COVID19

does the outcome from the rest 17 patients SIGNIFICANTLY harm this conclusion

(11/) the 2nd conclusion is “Measurement of efficacy will require ongoing randomized, placebo-controlled trials.” I will expect these evidence:

did adverse events DIRECTLY link to remdesivir and manageable?

Is the outcome SIGNIFICANTLY worse than expected without drug?

(13/) Note: if the conclusions are way beyond the CONTEXT could defined, Bye bye. e.g.,

CONTEXT:
potent inhibitor which kills pancreatic cancer cell line in petri dish.

CONCLUSION:
it cures pancreatic cancer.

(14/) With all these questions and QC, I start to read the rest of paper.

Note: put aside all the bias before proceeding. Only think about the CONTEXT and EVIDENCE. You may update the EVIDENCE CHAIN while reading.

(15/) If the Result has something you didn’t expect from the beginning (e.g., younger patients had higher mortality than older ones), think for a moment and read the “Discussion” to see if you find consensus.

Don't overinterpet the Discussion - it's just a place to share ideas.

(16/)
If anyone tried this protocol, but still thinks the Remdesivir paper is a shitty paper written by an evil pharma who only cares about $$$, and performed a political show by selecting data that only supports the conclusion, please let me know.

(17/) you may find yourself becoming a peer reviewer, which someone was supposed to do it for you. But this is something distinguish between scientific paper and "BREAKING NEWS" headline:

1. YOU READ BECAUSE YOU CARE.
2. News is data, but science is knowledge.

(18/) I am usually LEAST eager to know those least related to CONTEXT and EVIDENCE:

Author & affiliation
Funding
Journal name & impact factor

e.g., I don’t care whether author is 1 person or 10,000, small lab or big pharma, Nature or preprint

(19/) let me end this thread with a comparison

https://twitter.com/houndcl/status/1248736403711762433

Small update

When I am interested in a method paper, the above protocol doesn’t apply.

don’t just READ method paper (whether experimental or computational)

reproduce & adapt!

Most journals do better job nowadays enforcing the transparency of code, protocol and data source. This is phenomenal.

As reader, however, we also need to pay attention to adaptation problem.

Not all methods fit MY pipeline w/o any change. Spending some time digging out the hidden assumptions, major source of variation, applicability domain, etc is always worthwhile.