Thread by @IstvanHajnal

Took the effort to read the You Li et al. #COVID19 paper in The Lancet that was discussed on Belgian twitter (by @wduyck, @O_Thas, @CarlvKeirsbilck a.o. ) after @GMeyfroidt mentioned it on @deafspraaktv. 1/14
thelancet.com/journals/lanin…

It's certainly an interesting dataset they have assembled, but I'm less convinced about how they decided to model it. BTW, as usual, the supplementary material is more interesting than the actual article. 2/14

I don't think their setup allows for a causal interpretation, it's descriptive at best. They use a series of 28 separate linear models with the same set of explanatory variables, but each time a different dependent variable, i.e. the log of an R-ratio, one per day. 3/14

In other words, within each model they assume independence between countries (131) and between what they call phases (a time period when all 8 NPI's remained the same. 790 in total). This seems a very quistionable assumption to me. 4/14

Stacking the results of the different models, one for each day, rather than doing a proper time series analysis looks odd to me, but maybe that's how it's usually done in their field. 5/14

I would naturally be inclined to use a mixed model with this type of data. Interestingly the authors import the lmer-package (for Fitting Linear Mixed-Effects Models) in the #rstats-code, but they didn't use it as far I could see. 6/14

Either way, with their setup they don't seem to find much that is significant, eventhough the discussion could make you think otherwise. Maybe I'm misreading their charts. 7/14

As @wduck mentioned, they don't seem to be worried about multicollinearity. I would also think that multicollinearity is an issue, but then again, maybe that's why they don't find many significant effects (because under multic. the s.e. of the coefficients tend to be large) 8/14

It always helps to look at the (almost) raw data used in the analysis. Below is a representation of the raw data for BE and NL. They show a typical pattern of a sharp decline in R in March with Interventions starting roughly only in the second half of March 9/14

So, much of the decline is associated with no (listed) interventions at all. In reality it is obvious that by then people were already anticipating interventions and were listening to advice from epidemiologists, virologists a.s.o. 10/14

Another typical pattern you can see with many countries is that lifting of the interventions is associated with an increase in R. But for the Belgian school discussion it is vital to know that the analysis ends on 20/07/20. Schools were only effectively reopened in 09/20. 11/14

BTW, two other interesting countries I've lifted from their supplementary material are the US and Sweden. It is clear how the You Li et al approach goes wrong here (but then again that's just 2 cherry picked examples) 12/14

Just to be clear, I don't claim that interventions have no effect (I personally believe there is other evidence they do), but this study's setup fails to demonstrate that clearly. I give them some credit for illustrating the effect of relaxing restrictions 13/14

My conclusion is that this study, while interesting, can't be used in the current "school" discussion in Belgium. Not if you think schools should close again, not if you believe they should stay open (14/14).

@threadreaderapp unroll