Thread by @tylerblack32

THREAD: A look at a new study looking at school setting, pandemic, and behaviours in kids.

@JAMA_current article using a cohort design to look at the effects of the pandemic over time. They have an interesting statistical manipulation which I'll describe in a moment!

/1

Now very importantly, the authors acknowledge that the statistical manipulation wasnt enough! It falls short in a number of ways, so they caution very strongly against interpreting this as "causal evidence"

Let's look at what it shows and what its limitations are.

/2

First: what is this?

It is an ongoing survey that is being done of early learning that's been going since 2017. During the 2020-2021 school year, for kids 6-8, 405 parents were surveyed in 4 monthly surveys.

/3

Second: what was the question asked?

1) "Sometimes, children have periods when they do not behave like their usual selves. How has your child's behavior [sic, 🇺🇸] been in the past month?"

1) Much better
2) a little better
3) the same
4) a little worse
5) much worse

/4

2) a screen of 12 behavioural changes like tantrums, aggression.

3) for any hits, a scoring of "never, rarely, occasionally, all the time."

/5

Third: Who did they ask?

Parents. English speaking parents in Massacheusetts were surveyed. The survey was done online (a selection filter), voluntarily (necessary but again selects).

/6

You spoke of a "cool manipulation", what was it?

Due to how things went in MA, kids mostly went back to school. This allows for a cool peek into "what happens when kids back to school" but does come with problems (will discuss later)

/7

As you can see from the graph, there is VERY little moving "backwards" (more restrictive). 98% of the movement is "downhill" (remote to hybrid to in person).

/8

203 kids moved learning formats, it was, as the pandemic situation improved, basically "back to class." These parents of 6 to 8 year olds were being surveyed sequentially.

/9

Fourth: What did they find? (positive=worse)

Remember the questions asked.

General: a scale from 1-5 "much better to much worse"

Maladaptive: 0-12 item "checklist"

Dysregulation: average of 11 maladaptive behaviours scored from 1 (never) to 4 (all the time)

/10

And, as such, they found differences (in the worse direction) on all scales comparing "remote" to "in person" and "hybrid," and scales comparing "hybrid" to "in person".

/11

So, why doesn't this show cause? (and why did the authors take such care to ensure that people would not associated it with cause?)

Well, because this is Massachusetts during the pandemic. I've labelled the 4 waves of the study.

/12

Because the "moves" were almost entirely in one direction, and the pandemic itself and its effects on Massachu..set..tsians? was in one direction, there is NO separation between pandemic and school setting.

/13

A very small handful of kids went "in person" to "remote", for example. By my count, about 7, for the comparison of "in person vs remote", whereas it looks to me that about (guessing) 55 or so went from "remote" to "in person."

/14

Oddly (I don't know why), this paper does not look at, address, or report kids who DIDNT change condition (remote 1 - remote 2, hybrid 1-hybrid 2, in-person 1 to in-person 2) & there were MANY more kids in that set than switched.

I've drawn boxes around these huge sets

/15

This, (i'm guessing), would help address, at the very least, an estimate of the "time effect" that limits this study. Maybe its brevity, but I would say, if those bands show similar changes then i think we kind of have our answer?

/16

But aside from the "time confound" that the authors recognize as a major limitation (*why title it this way?!*), there are also some pretty big limitations to this study.

/18

First limitation, no estimate of effect size, but eyeballing it, it's not particularly impressive. I suspect if a cohens d was used for within-individual differences, we might be looking at 0.05-0.3ish sizes.

Like, the largest difference I can see, is the "general scale"

/19

Remember, this is a 5 point scale.

"hows your childs behaviour in the past month?"

1 - much better
1.5
2 - little better
2.5
3 - same
3.5
4 - little worse
4.5
5 - a lot worse.

0.5 is one half step on that scale.

/20

Also, it's a "within-kid difference" measure, so if last month "much better" then next month "a little better", measure is +1 WORSE, even though this sounds to me like continued improvement (if it was a little better than last month & last month was better...)

/21

0.5 is probably a small effect size but not negligible, but if the truth is within the CI it could be as low as 0.3, which i would suggest is quite a tiny difference on this scale.

/22

The effect sizes then plummet. on the "behavioural checklist", it's out of 12 yes/no behaviours. Changing by 0.5 on average is not particularly impressive. Out of 12 possible behaviours, remote vs inperson changed by half of one "yes no", with no quantification.

/23

For the "dysregulated," remember it was 11 behaviours being scored 1 never to 4 always. changing by at most 0.1 on average is statistically significant but clinically meaningless.

/24

Second major limitation: parent survey.

It has been well established that parent surveys are OK (can't think of better survey design for <10y in pandemic - not during a pandemic) but show extremely poor corelation with observed behaviours (0.06)

ncbi.nlm.nih.gov/pmc/articles/P…

/25

This maybe stings for parents who think they know their kids best, but take heart, there is no judgment here! How we see our kids doing is how we see ourselves doing, our issues doing, our world doing, and our peers doing. Nothing beats direct observation.

/26

Now, there were some controls here on this. Authors state they controlled for mental health but used the extremely insufficient PHQ-4. Because its teeny tiny, I can post the whoooooole thing here. Not exactly an intricate measure of parental wellbeing. Each a 4 point scale.
/27

Overall, I would say that surveying parents during a pandemic (especially with a significant moral panic of school closures=bad for kids, thanks media), has real limitations.

/28

Finally, it's important to recognize that 6-8 year olds are a tight group, and this is a very select finding. These results are not generalizable to ALL KIDS or even middle school kids. 6-8 is early school. Same with MA, English speaking, etc. Careful with generalizing.

/29

My strongest request (and I will email the authors), is i'd love to see t-1 t-2 comparisons for kids who DIDNT switch condition, and of course knowing what the average scores were is important not just the change in scores.

/30

For strengths of the study:

a) It's a pre-established cohort. 4th year of the study. Would be awesome to see how year 4 compared to year 1 of the study too, just in terms of raw numbers.

b) It's statistically significant (though many of these changes seem clinically small)
/31

c) cool design. If more kids went "backwards," some REALLY good statistics could have been done, but it would be weird if remote increased if pandemic decreased, wouldn't it?

d) this is my favourite type of flow graph. Love it! But more helpful to show n in each connection.
/32

No doubt the authors warnings will be ignored and some media/advocates will use this as evidence that "remote learning causes behavioural problems in kids." And i'll be endlessly spammed this article even though I spent >30 tweets carefully going through it :P

/33

My biggest bone to pick. If the conclusion reads as it does, why does the title read as it does? I know the title is more sexy and probably convinced @JAMA_current to publish. But this is 2022 in the middle of a misinformation war.

/34

The authors, @JAMA_current , & everyone in the known universe knows that people are misusing "latest science" in information wars/strategies. By making the title abt "School learning format and childrens behaviours", & expecting the reader to read the print of the abstract...
/35

Well, honestly, its just bad science communication. The authors COMPLETELY KNOW that the kids mostly switched downhill and due to the pandemic changes too, they had NO ABILITY to separate learning format from pandemic.

/36

I hope this choice was naive and not shrewd, because if this paper gets traction (and i am rather sure it will), it will almost entirely be used in misinformation battles claiming that it is proof (or even slightly strong evidence) of cause.

/37

In reality, it adds to a number of studies that shows that kids have, typically, done better as the pandemic has done better and done worse as the pandemic has done worse. This study TYPE has the ability, but in reality couldn't separate school from pandemic.

ERRATUM: Tweet 5 is incorrect. The final item "dysregulated" was not only asked of "positive items," it's a 1-4 scale (never to always) of 11 behaviours of concern, regardless if the previous scale identified them in the checklist... I think!

ADDITION:

another strength (sorry, this might be not obvious to all).

It's longitudinal. It's still longitudinal surveys cross sectionally (the sum of multiple correlations doesn't = causative, but i usually accept "leans") , but its better than cross sectional for sure!!

2nd addition:

I have no idea what the computational cohens d would be. because its within-person, its hard. For statistics types I recommend the discussion: jakewestfall.org/blog/index.php…

I personally agree that cohens d is best and likely represents the clinical "effect size"