,
12 tweets,
4 min read
Read on Twitter
Large quantities of data = High quality of science. My new Nature Human Behaviour paper published today with @ShuhBillSkee shows that this could not be further from the truth. nature.com/articles/s4156… (1/10) 
For our work we focus on the association between digital technology use and teen well-being, which has been getting a lot of press lately. Too much, if you ask me. (2/10)
Some days people find negative associations, some days positive, some days none at all. Why is there no consistency in scientific findings? (3/10)
Many of these studies examine openly-available large-scale datasets with many thousands of variables and participants. We found that almost every paper analyses such data slightly differently, e.g. in how they define well-being (4/10)
But also in how they define digital technology use, what control measures they use and what models they run. We found that scientists could have analysed the data in over a trillion different ways - just to answer 1 simple research question! (5/10)
We used Specification Curve Analysis (an innovative statistical technique), to examine how much impact this potential for analytical flexibility has on the scientific conclusions reached in a resulting paper. (6/10)
Short answer: A huge impact! Running over 20,000 analyses on 3 different datasets we found that scientists could have written 1000s of papers finding positive, negative and non-significant associations between well-being and digital tech - it all depends on the analysis (7/10)
For example, defining digital technology use in different ways affects the results found (8/10)
Or… whether you include good quality controls in the analysis or not. As you see, not having proper controls makes the association between digital technology use and well-being a lot more negative. (9/10)
The paper powerfully visualises that without pre-registering analysis plans beforehand, analytical bias can allow researchers to tell almost any story with powerful data resources. Is this really what science is supposed to be like? (10/10)
Read more about the thoughts behind the paper here: go.nature.com/2RDy1ci
Open Access version can be found on amyorben.com!
