,
11 tweets,
2 min read
Read on Twitter
Starting now @McGillPsych "Solving the replication crisis with multilevel modeling" by @StatModeling (Andrew Gelman), my second ever attempt at live tweeting!
Most people in the room have heard of the replication crisis--part of the crisis is that various well known studies didn't replicate and the other part of the problem is studies we know wouldn't replicate, so we don't even try.
So what is the problem: the problem is that if you get statistical significance you can assume something is real and move on, but then you realize you are wandering in a loop where what you thought was real has disappeared. What is the solution?
In stats there is the armstrong principle, if you promise something you can't deliver, you are motivated to lie. We are often in this situation, study results have to be significant to be published.
Incremental improvements can not be studied using conventional statistical methods, if you have smaller effects you need more people and you can't test small improvements, you are setting yourself to try to prove something.
If you have a small effect size you can't be as certain of your effect, you can't get the proof you want. But what is the solution? Three kinds of solutions (serious ones not to stop criticizing other people's work): procedural solutions, design and data collection, data analysis
Procedures are things that are important--teaching, incentives, changing the rules, not his area of expertise. Then there is design/data collection and data analysis, design is more important because it determines what data you have to analyze
Data analysis is important, allows you to get more information out of your data, good data analysis may suggest what data you need to go next, run a robustness study, not just do a better analysis, but use multilevel models to combine data from multiple studies
practical advice: better measurements. Better measurement isn't free, might need to ask more questions, but at the sacrifice of being able to collect more measures. Collect more measurements, you can substitute more people for good measurement, you can still have measurement bias
Attitude is that if you found the effect you don't have a measurement problem, it isn't true, it is a trap. Small standard errors don't ensure a lack of measurement error. We need multiple measures, more within person designs, accept more uncertainty in our results
Conclusion: there is a replication crisis, multilevel modeling is a solution, if helps us analyze data better and motivates us to collect better data (combine data from multiple studies, more within person designs, more and better measures).
