Thread by @arjunmanrai

(Thread) 2019 problem: you have a big dataset and everything is significant after Bonferroni. What's spurious; what's real? How do you prioritize? Happy to share my take w/ John Ioannidis & @chiragjp out now in the American Journal of Epidemiology: academic.oup.com/aje/advance-ar…

1/n 👇

Some background: Inspired by Genome Wide Association Studies (GWASs), 'X-WASs' have emerged as systematic, agnostic, and reproducible approaches to discovery with non-genetic data (e.g. X = Environment, Medication, Phenotypes...etc.) For ex., the EWAS by @chiragjp et al. 2/n

A practical challenge today (with datasets in the hundreds of thousands or millions of subjects) is that often there are hundreds/thousands of significant signals, many with minuscule p-values and many with modest effect sizes. 3/n

Non-genetic X has additional challenges over GWAS. For example, many exposures are often difficult to measure, time-varying, and densely connected (see ex. below). For more: stm.sciencemag.org/content/1/7/7p… or ncbi.nlm.nih.gov/pubmed/25592584 4/n

In GWAS, we visualize confounding (e.g. due to stratification) as deviation from the null in a Q-Q plot (ex. below from Price et al. 2010). We often measure confounding with λ, the 'genomic inflation factor' / 'genomic control' (defined below by Price), which should be ~1. 5/n

In our article, we propose an analogous 'X-wide inflation factor' for non-genetic X, denoted λ_X: 6/n

We computed the X-wide inflation factor (λ_X) for previously published XWASs and observed that it tends to be much greater than 1. 7/n

There are key implications for non-genetic association studies incl. likely massive residual confounding in some cases. We will need to be nuanced with findings from large country-scale biobank studies going forward given the relationship between λ_X and sample size. 8/n

A number of approaches to prioritize signals among the many significant findings may help, each with its own strengths and limitations. Here are a few we suggest: 9/n