Thread by @GidMK

Something that I keep seeing pop up is the idea that meta-analysis somehow eliminates issues with the underlying research

This is just confusingly incorrect

Meta-analysis really is just a simple statistical aggregation of results. Indeed, the most basic way to meta-analyze studies is just to use a mean or median of their point-estimates

But simple means/medians can be misleading - we don't really want a study of 10,000 to have the same weighting as a study of 10, but that's what a simple average provides

When we run a meta-analytic model, what we are actually doing is generating a WEIGHTED mean/median, and confidence interval. Essentially, we take all of the means and SEs, and based on that our model weights them with bigger studies generally contributing more to the model

You'll notice I've said nothing about the underlying quality of the evidence

That's because META-ANALYSIS HAS NOTHING TO DO WITH QUALITY OF EVIDENCE

You can throw anything into a meta-analysis model. Here's a model I just ran on the ratio of hosting to participating in the summer Olympics. This is meaningless!

We tend to put meta-analyses on a pedestal, but the fact is that statistically aggregating evidence is a total waste of time if that evidence is all bad

This recent Cochrane review is a perfect example - they looked at the evidence for ivermectin for COVID-19, but because most of it was terrible they only included a few studies in their model

https://twitter.com/GidMK/status/1420340231786549253?s=20

This is also where the phrase "garbage in, garbage out" comes from. If your meta-analysis includes numbers from studies that are terrible, the final point estimate is as meaningless as my graph above on the Olympics 🤷‍♂️

Some people think meta-analysis is impressive because it involves fancy statistical software, but it's entirely possible to implement a Dersimonian-Laird inverse-variance model in Excel with a stats textbook and a few hours of time