Thread by @cremieuxrecueil

The study they're reporting on is not causally informative and does not indicate actual reasons to be concerned.

The reasons for this are very simple, as I'll show below.

🧵

https://twitter.com/CBSEveningNews/status/1823497830721348013

First, sampling.

If I want to do a study on Holocaust survivors and I go and seek out people who survived it, I am looking for a select sample.

If, instead, I look in datasets that were sampled without respect to Holocaust survival and find survivors, my sample is nonselect.

Why does this matter?

Select respondents differ from nonselect ones because they elect to be sampled or because I was able to find them by virtue of something that differentiates them from the population.

For example, my Holocaust survivors might be part of a support group.

It might be that only the ones who suffer the most go to support groups. If I sample from the support group, I could contaminate my results because I'm sampling super-sufferers, not the norm.

That's actually a problem and I've pointed to it before.

https://x.com/cremieuxrecueil/status/1726700827593888243

The study CBS is referring to was based on a moderate-sized (n = 315 parents) convenience sample.

What happens when you use a large, population-representative sample to look at this screen time/tablets question?

Well, usually nothing more than chance:

https://x.com/cremieuxrecueil/status/1726155119291707476

Second, causality.

The study in question at best merely associated tablet use with outbursts of anger, it didn't provide meaningful causal evidence to explain this association.

When the relationship screen time has with internalizing problems and attention problems has been looked at with genetic data, the whole relationship has been found to be due to genetic confounding.

https://x.com/cremieuxrecueil/status/1693830911140258002

Accordingly, to conclude anything about the causality of the relationship between table use or screen time or whatever and kids' behavioral problems, we need to do things differently: we need large, representative samples with causal designs.

But wait, you might say: They had a random-intercepts cross-lagged panel model, so they could make causal inferences!

True, but the result is embarrassing, and especially so given the stronger literature on this finds nothing. They had two focal p-values: 0.045 and 0.040.

p-values this marginal were almost-certainly not found by mistake. They represent p-hacking, and if you're willing to talk to the press about p-hacked findings, maybe we should question your other work too.

Here's the study:

And an article on marginal p-values: jamanetwork.com/journals/jamap…
cremieux.xyz/p/ranking-fiel…

To make the p-hacking clearer, consider what the authors discussed in the abstract versus the results.

They generally found nonsignificant paths, but only discussed the two significant ones. Unless they're proposing age alters the nature of the causal relationship, that's no good. The fact that they only talked about the p < 0.05 results even though the others weren't distinguishable is just odd. It would be a weak result either way (especially for a reason I'm about to say), but the way they decided to write about this suggests they really wanted to get published, not to deliver the null they really found.

Also, unmentioned in the abstract: They had lots of attrition. 16% of the sample was lost by T2, and by T3, they had lost a third. Without imputation, that p = 0.04 is almost-certainly going >0.05, and it might even be >0.05 just using a different imputation method than FIML.

After re-estimating the model from the correlation matrix, I can confidently say that the highlighted significant results (p = 0.04/0.045) are down to the use of FIML.

The method of p-hacking was easily identifiable.

Good job accepting this, @JAMAPediatrics!