Thread by @NoahHaber

It's been about a week since we've had developments in the insane story of the nutritional epi Willet et al., vs. Johnston et al. "controversy" over meat studies.

It's easy to think about it as a personal showdown between people. But it's so, so much more. Thread.

1/n

To get us started, you read this: jamanetwork.com/journals/jama/…,

and then skim this: texasmonthly.com/article/texas-…

I'll wait.

Ugh fine I'll summarize (but you really should read them)

2/n

Nutritional epi has a reputation as the poster child for sloppy methods which give meaningless and misleading results. That's not ALWAYS true, but the stereotype doesn't come from nowhere.

The current controversy over meat highlights exactly how and why perfectly.

3/n

As usual, it's all about causal inference. No one cares if people who eat more eggs get Alzheimer's more; what we ACTUALLY care about is whether eating more eggs results in more/less Alzheimer's.

The difference between those two things is EVERYTHING.

4/n

In nutrition, we can't really do studies that get at causation in a satisfactory way. Randomization and long-term follow up is impractical (but maybe more practical than many believe), "control" style causal methods are confounded, and instrument opportunities are rare.

5/n

The available study designs and methods are often barely better than rolling dice, or worse.

At the same time, people really really want to know what they should and shouldn't eat. Any "evidence" would be HUGE! So we have lots and lots of weak and misleading studies.

6/n

Enter: Walter Willett and Frank Hu, two of the most cited researchers in all of health research (hsph.harvard.edu/nutrition/2018…).

Willett was chair of the Harvard Nutrition dept. for many years. There was a mini festival in his honor when he retired.

7/n

To put it mildly, a whole lot of it doesn't pass the sniff test for methodological rigor. A lot a lot of it. And of COURSE it doesn't; as above, it's nearly impossible to do rigorous, conclusive studies for these kinds of things.

But it certainly doesn't stop those pubs.

8/n

Willett and Hu have a tonne of (weak) research which does a lot of "standard" winking and nodding about red meat being bad for you. It's never actually causally identified (because it can't be), but they've staked a LOT of reputation on that.

9/n

Willett has a bit of a history of getting personal, often writing incendiary articles accusing people whose studies disagree with his own of having weak/biased methods (and ignoring that his own are often weak/biased for the same reasons).

bostonglobe.com/magazine/2013/…

10/n

Enter: Bradley Johnston. Johnston has an interesting twist here. His work is all about relying on more rigorous studies for the best evidence (mostly RCTs). And that seems like it makes sense on the surface, but doesn't really hold up either.

11/n

He looks at the RCTs, finds that there isn't causal evidence for negative health impacts of meat, and declares that meat is safe.

But that's ALSO wrong, for a different reason.

RCTs are great when you can do them, but we only have really really limited RCTs for meat.

12/n

What we have is short term, limited settings, limited outcomes, etc. If there is a substantial impact of meat on health, it's all in the long run, which we didn't can't measure.

And there's the second mistake: absence of evidence is not evidence for absence.

13/n

We are now at tweet 14, and notice that I haven't mentioned the one thing everyone's talking about even once: $.

All of the folks here have a lot of industry funding. Academic funding is extremely limited, industry $ is not.

But you shouldn't think of this as "corruption"

14/n

All the folks here believe they are doing the right thing with the best methods available. The $ influence is more selectively amplifying voices / study choices than it is changing individual researcher's outcomes much.

15/n

But when the methods are highly cited publicly engaged dice rolls, there's a whole lot of inventive to fund your particular preferred kind of weighted dice.

The story here about conflicts of interest is a bit of a distraction in a way.

16/n

In the end, the funding problems are a consequence of a much bigger, more fundamental problem in the field of nutritional epi: accepting weak methods as the acceptable standard creates all kinds of opportunities for all kinds of bad behavior.

17/n

This story is about the problems in modern research science, and not so much about the influence of money or personalities.

It's the reason we in the research community need to get our shit together. If we don't, this will keep happening.

18/18

Since this blew up a little, FOOTNOTES AND CORRECTIONS!

1) "Declared safe": Johnston et al., went well beyond just saying there is not enough evidence, and positively recommends "adults continue current unprocessed red meat consumption"

1 cont) Lots of coded language. "Technically we never said safe" is equivalent to "Technically we never said cause."

2) "All of the folks here have a lot of industry funding." I'll walk this one back a little. I don't know that they are currently directly industry funded.

2 cont) Other ways that industry funding can work in here, but maybe not as directly as I stated.

3) I never actually summarized what happened in the last few weeks. Whoops. If only there was some kind of "article" that one could "read" to absorb information...

4) Don't want to pick on people (handle redacted to avoid brigading), but this tweet ironically (and spectacularly) highlights what happens when people perceive low standards for strength of evidence in epi.

5) A guideline based on evidence alone should have simply stated that there is no evidence, and therefore make NO recommendations.

A better guideline, however, would have incorporated the best available theory and expert opinion where evidence is lacking.