Thread by @GidMK

The wonderful world of nutritional epidemiology

Two studies, months apart. Both looked at egg consumption. Completely opposite results

What's happening here?

Now, there's a lot of criticism about the media reporting and content of both of these studies. @dailyzad's blog on the recent one is here lesslikely.com/nutrition/eggs… and I've written about the older one here medium.com/@gidmk/eggs-wo…

And while there may be issues with the studies, what I wanted to focus on today was something that is often missed in the discussion of these epi studies: generalizability

Let's kick off with a question: are studies with bigger sample sizes MORE or LESS generalizable?

This is important because it's a discussion I have professionally ALL THE TIME

It's also something that comes up on twitter constantly

The question really comes down to whether increasing the sample size of a study reduces the biases in your data collection

Let me explain

An example from my own work in diabetes is looking at prevalence. We want to know how many people have diabetes, but it's very hard to test enough people in a rigorous way to be sure of our estimates

Instead, we look at a sample of people who have been tested. Let's say we take every person who has attended one of 10 GP clinics, and look at their diabetes test results

That's a reasonable number of results, say 50,000

So we've got data from a group of people who have attended some GPs, and we get a figure about how many of them have diabetes

Is this figure generalizable to the population (i.e. can we use it to estimate the proportion of people who actually have diabetes?)

What would happen if we then added another 100 GP clinics, or 500,000 patients, to the sample?

Would it be more or less generalizable?

Now this is a lot easier than that first question. Collecting data from people who visit their GP clinics introduces obvious biases into the equation

They are more likely to be sick, are usually older, often richer than the general population

Etc etc etc

GETTING MORE RESULTS DOESN'T NECESSARILY REDUCE THE IMPACT OF THESE BIASES

To put it another way, our bigger sample is just as problematic as our smaller sample in key ways

We've got more people, which makes the estimate more precise, but it doesn't make it more generalizable to the population

In other words, we have a very good figure for how many people have diabetes in the group of people who visit their GP clinics, but NOT for the general population because of the bias in our sample!

This brings us back to the original question

The answer? Bigger sample size makes NO DIFFERENCE to generalizability. It's the QUALITY not QUANTITY of the sample (mostly)

https://twitter.com/GidMK/status/1110729194680381440

Note: it ~can~ make a difference in smaller sample sizes. We're talking here about the difference between 1,000 and 100,000, not 10 and 100

So let's go back to our original two nutritional epidemiology studies. Both were huge (n=50,000 and 500,000)

BUT they had very different populations

The recent study looked at a group of people living in North America. The older study was on adults living in China

These are very different

We can't expect the results of Chinese people, even if the sample size is HUGE, to directly generalize to people living in the USA

These two regions of the world have vastly different food intakes, very different cultures, and this means that egg eating has different meanings and connotations in both places

This was actually central to the hypothesis of the Chinese study. The researchers argued that Chinese eating habits were different, and so the egg consumption should be studied to see if it was beneficial despite previous studies in other populations showing harm

In fact, while the two studies APPEAR contradictory, it's likely that this is, in part, context-based

Chinese people who eat more eggs are healthier

Americans who eat more eggs are less healthy

BOTH OF THESE THINGS CAN BE TRUE

TL:DR - two studies appear contradictory (eggs good or bad?)

Realistically, it's likely just the differences between the settings for the studies

Bottom line, bigger studies are not always better it's more complicated than that

P.s. the REASON that egg eating is associated with good or bad health is, in my opinion, more complex than either of these studies suggests

My personal argument is that it's ~almost certain~ that both of these studies are using egg eating as a marker for some sort of social influence that is really causing the heart disease

So people in the US who eat more eggs are probably less well off in many ways, with the inverse being true in China