Profile picture
, 12 tweets, 4 min read Read on Twitter
Large quantities of data = High quality of science. My new Nature Human Behaviour paper published today with @ShuhBillSkee shows that this could not be further from the truth.… (1/10)
For our work we focus on the association between digital technology use and teen well-being, which has been getting a lot of press lately. Too much, if you ask me. (2/10)
Some days people find negative associations, some days positive, some days none at all. Why is there no consistency in scientific findings? (3/10)
Many of these studies examine openly-available large-scale datasets with many thousands of variables and participants. We found that almost every paper analyses such data slightly differently, e.g. in how they define well-being (4/10)
But also in how they define digital technology use, what control measures they use and what models they run. We found that scientists could have analysed the data in over a trillion different ways - just to answer 1 simple research question! (5/10)
We used Specification Curve Analysis (an innovative statistical technique), to examine how much impact this potential for analytical flexibility has on the scientific conclusions reached in a resulting paper. (6/10)
Short answer: A huge impact! Running over 20,000 analyses on 3 different datasets we found that scientists could have written 1000s of papers finding positive, negative and non-significant associations between well-being and digital tech - it all depends on the analysis (7/10)
For example, defining digital technology use in different ways affects the results found (8/10)
Or… whether you include good quality controls in the analysis or not. As you see, not having proper controls makes the association between digital technology use and well-being a lot more negative. (9/10)
The paper powerfully visualises that without pre-registering analysis plans beforehand, analytical bias can allow researchers to tell almost any story with powerful data resources. Is this really what science is supposed to be like? (10/10)
Read more about the thoughts behind the paper here:
Open Access version can be found on!
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Amy Orben
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!