Tweetorial: Do early life non-cognitive skills affect academic, psychosocial, cognitive and health outcomes? 1/16 @Lisa_Smithers @chittles74 @mendel_random nature.com/articles/s4156…
It is often claimed that non-cognitive skills such as character traits like grit and personality are critically important for child development. These traits are thought to be more malleable than intelligence. However, it's not clear what evidence supports these claims. 2/16
We systematically reviewed the literature. First this literature is huge! We screened 9553 studies, reviewed 554 eligible papers, and interpreted results from 222 papers. We meta-analysed the results across all the studies for each outcome. 3/16
What did we find? There was some evidence from the meta-analysis of effects on externalising behaviours and social skills. The forest plot shows RCT and quasi-experimental studies above, and observational studies below. 4/16 
However, this doesn't allow for small study and publication bias. This can occur if the more positive and statistically significant results are more likely to be published. We investigated this using funnel plots. 5/16
If estimates had a similar likelihood of being published regardless of their effect size or statistical significance, you'd expect a funnel plot to be symmetric. If "significant effects" are more likely to be published the plot will be asymmetric ageconsearch.umn.edu/bitstream/1162… 6/16
What we found was rather interesting. All the outcomes showed substantial evidence of bias. These charts show a funnel plot of the effects of non-cognitive skills on academic achievement. 7/16
These funnel plots are for psychosocial outcomes. Again, substantial evidence of bias. 8/16
The same plots for cognitive and language development. Again highly asymmetric funnel plots. 9/16
Finally, there were insufficient RCT and quasi-experimental intervention studies for health outcomes for a meaningful funnel plot, but there were more than enough observational studies. Again, the plots suggest major bias. 10/16
Has anyone more one sided funnel plots than that? One explanation for these results is that negative and null results are not published. Our meta-analysed results could be largely explained by small study and publication bias. 11/16
Overall the quality of studies was low - e.g. a third of observational studies made no attempt to control for confounders. The follow up was generally extremely short, making it difficult or impossible to assess long term effects. 12/16
So what can we conclude from 222 studies? Do non-cog skills affect psychosocial, cognitive and health outcomes? Maybe, but overall the evidence published in literature is deeply flawed and far from compelling. If non-cog skills do matter, we don't currently have the proof. 13/16
How can we make progress on this hypothesis? Fewer bigger studies, with pre-specified analysis plans and longer follow-up, more credible study design to address clear causal questions. 14/16
Caveat - any mistakes above are mine, all the hard work - seriously, this took years of their time, was done by @Lisa_Smithers, Alyssa Sawyer, @chittles74, and John Lynch. 15/16
Do read the paper - v. happy to send the PDF, but all the important information is in the (free) appendix. Happy to answer questions! 16/16 nature.com/articles/s4156…
