Many researchers make the same mistake in interpreting their findings that none other than Donald Trump recently made when talking about COVID and masks:
That is, they assume that explaining variance is a good measure of causal impact. “Explaining variance” comes up over and over in widely different scholarly literatures, including which teachers are effective, which characteristics of people are heritable, and more.
But as Richard Berk explains in his book Regression Analysis: A Constructive Critique, "Contributions to explained variance for different predictors do not represent the causal importance of a variable."
Working out the math would be difficult in a tweet, but here’s a commonsense explanation:
Imagine that when people get MD-level training in oncology, they are 50 times better at improving cancer mortality than people without medical degrees who run naturopathic websites.
If you did a study of oncology effectiveness just in the first group, you would find that having MD training in oncology explains zero percent of the variance in cancer patient outcomes.
That’s because all the people you’re studying have MD training in oncology.

Thus, whatever explains the variance in their patients’ outcomes, it can’t possibly be *that*.
But zero explained variance doesn’t mean it is useless to have MD training in oncology.
As I say, some scholars seem to make this mistake. But so does Donald Trump along with other mask skeptics.

In a town hall on October 15, 2020, Donald Trump was questioned about the use of masks to prevent COVID19 transmission.
A telling part of the exchange:…

Trump: “But just the other day, they came out with a statement that 85% of the people that wear masks catch it.”
Interviewer: “It didn’t say that. I know that study.”

Trump: “That’s what I heard, and that’s what I saw.”
So, here’s what the study in question actually did:…

They found 154 patients who tested positive for COVID, and compared them to 160 patients who had similar symptoms but a negative COVID test.
In the COVID group, 85% said they had often or always worn masks in the prior two weeks, compared to 88.7% of the other group.
So Trump reversed the conditional probability here: The study didn’t ask “how many people who wear masks get COVID,” but “how many of these COVID patients wore masks.”
These are very different questions. If 85% of the people who wear masks get COVID, then we should all have COVID by now.
Let’s take a charitable interpretation of what Trump might have been trying to say:
From this data, it does look like mask-wearing would explain very little of the variation in who gets COVID.

Which mask skeptics would trumpet: “See, both COVID and non-COVID people wore masks! Wearing a mask didn’t explain why they did or didn’t get COVID!”
But so what?

For the sake of argument, let’s say that the rate of getting COVID is 30% without masks and 3% with masks. If that were the case, masks would reduce the risk by 90%.
Now, if nearly everyone wears masks, then whatever explains the difference between the 3% of people get COVID versus the 97% who don’t, it couldn’t be the masks. It would have to be some other factor.
But the lack of explained variance in no way changes the fact that, in this hypothetical, widespread mask-wearing reduces everyone’s risk by 90%.
Moral: You shouldn’t look to Trump to interpret scientific evidence correctly, but neither should you look to researchers who put too much weight on “explained variance.”

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Stuart Buck

Stuart Buck Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @stuartbuck1

18 Jul
Cultural innovation is speeding up due to technology and social media.
I have a teenage daughter who regularly utters words, sentences, and even whole paragraphs that I find completely indecipherable, and it’s always about some Gen Z cultural phenomenon.
Just one quick example, which will take some background:

There’s something called the Accent Challenge on TikTok, where people read lists of words in different accents:…
Read 13 tweets
17 Dec 19
Trust no one: A National Academy of Medicine special.

I've been looking into a recent NAM report on opioid use disorder, titled "Medications for Opioid Use Disorder Save Lives." A simple declaratory title.…
But I was interested in some of the claims in this paragraph on page 39:
Check out the first claim: medication reduces mortality by up to 50 percent. First citation: Cicero et al., 2014.

Here's that citation:…

Note the title: "Factors contributing to the rise of buprenorphine misuse: 2008–2013."
Read 18 tweets
24 Sep 19
In funding many research projects, I've often advised researchers to be sure to correct for multiple comparisons.

But I'm having second thoughts. Interested in reactions from #econtwitter and stats folks.
The goal is to penalize p-hacking wherein researchers compare lots of outcomes, subgroups, etc. The method is to penalize the p-value metric (Bonferroni, Holm, Westfall-Young, Benjamini-Hochberg, etc., all work by lowering p-values in some way).
But maybe it's better to do:
Read 19 tweets
17 Jun 19
Important new paper from Larry Orr, Rob Olsen, @lizstuartdc, and others.…
In thinking about evidence-based policy, policymakers might often wonder, "Will the results of one evaluation/RCT apply to my jurisdiction?"

Orr et al. look at three RCTs in education (one on 36 charter schools, one on ed tech in 132 schools, and one on 84 Head Start centers).
They ask, "Could you have predicted the results in a given school/center, by relying on the results from all the other schools/centers in that same RCT?"

Answer: not very well.
Read 7 tweets
10 Jul 18
I was going to write up a response to this PNAS piece: Redish et al., “Reproducibility failures are essential to scientific inquiry.”…
But I realized that 1) PNAS might not publish it; 2) Even if they did, it would take a long time; 3) Posting my thoughts on Twitter would probably reach more people (about 2,000 people minimum, and a lot more depending on retweets).
So here goes. I’ll start with a summary of what Redish et al. argue. They say that “current discussions of the reproducibility crisis overlook the essential role that failures of reproducibility play in scientific inquiry.”
Read 24 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!