Thread by @MariaGlymour

I hope everyone reads the results in this paper but ignores the conclusions, since conclusions do not seem to reflect the results Batty et al: bmj.com/content/368/bm… @PWGTennant @epi_kerrykeyes @EpiEllie @EpidByDesign @MarcusMunafo @MikaKivimaki 1/n

Their question is super important: can we generalize from highly selected samples (HSS) eg UK Biobank (UK) to populations? HSS are much cheaper than representative samples (and in general, high response rates are expensive to achieve), but … 2/n.

Causal estimates in highly selected samples can differ from truth in populations if (1) effects differ in the people who participate vs those who did or (2) selective participation creates spurious associations so we don’t get the right answer even for those who participated! 3/n

We know bias is possible but we don't know if it's common in UKB. Stats tools can help generalize from highly selected samples to populations, but only if we have the right measures & enough diversity in the original HSS. So Batty paper is very important. 4/n

There's a social justice angle here: highly selected samples are usually differentially white & high income. Do we need "costly" diversity in research to learn about population health effects? Batty’s results say “yes”! Evidence from HSS are not reliable w/o correction. 5/n

They compare risk factor effect estimates from @ukbiobank to the same estimates from Health Survey of England (HSE), a representative sample. In Fig 1, for 5/7 comparisons, they find statistically significant differences; 5 HRs were more than 15% different. 6/n

Fig 2, 3/10 comparisons statistically significant diffs between UKB and HSE, all large differences, e.g. HbA1c predicting CVD death, HR=3.86 in UKB, HR=1.98 in HSE for doubling of HbA1c. 7/n

The good news from Batty: for every comparison they present the effect estimates were the same sign. The bad news: if you used the UKB estimates to make public health decision or prioritization, you’d often make the wrong decision. 8/n

Example: which might have bigger benefits as a prevention target, phys activity (UKB HR=3.4, HSE HR=2.3) or education (UKB HR=1.6, HSE HR=1.9)? In UKB, phys ax looks most important by far (3.4 vs 1.6!); in HSE, different picture (2.3 vs 1.9) given low education is more common 9/n

How about targeting HbA1c control (UKB HR=3.9, HSE HR=2.0 per doubling) versus hypertension (UKB HR=1.9, HSE HR=2.6)? In UKB looks like HBa1C likely dominates (3.9 vs 1.9!) but likely reversed in HSE (2.0 vs 2.6). 10/n

So, you generally get the “right” answer in UKB if you only need to know the sign of the effect (good or bad). If you care about the magnitude, or the relative importance of different risk factors, you may or may not get the right answer in UKB. It's hit or miss. 11/n

This doesn’t mean UKB is not useful: it is an invaluable resource. But we need to evaluate generalizability on a case-by-case basis. Need to see how modern stat methods to improve generalizability perform in UKB vs HSE, which they didn’t apply here. 12/n

My bottom line: nice results, unfounded conclusion. 13/n

If you care about selection bias and generalizability, check out: Keyes & Westreich: doi.org/10.1016/S0140-…
Westreich: doi.org/10.1093/aje/kw…
Gleason: doi.org/10.1016/j.jalz…
Bareinboim: jstor.org/stable/43288500 14/14

And to the people cringing about the statistical significance language above-technically this is evaluating an interaction (between study sample and risk factor) so even epi would grudgingly accept. 15/14