Can machine learning outperform baseline logistic regression for predicting complex social phenomena? Many prominent papers have claimed highly accurate civil war prediction. In a systematic review, @sayashk and I find these claims invalid due to errors. reproducible.cs.princeton.edu 
We are not political scientists and the main point of our paper is not about civil war. Rather, we want to sound the alarm about an oncoming wave of reproducibility crises and overoptimism across many scientific fields adopting machine learning methods. We have an ongoing list:
Incidentally, we learned about one of the systematic surveys in the above list because it found pitfalls in a paper coauthored by me. Yup, even researchers whose schtick is skepticism of AI/ML are prone to overoptimism when they use ML methods. Such is the allure of AI.
We don’t attribute reproducibility failures to the carelessness of individual researchers. We view applied-ML research as methodologically immature and far more prone to pitfalls than applied-stats research. For now, we must treat applied-ML research findings with caution.
Last year I offered a course on limits to prediction with @msalganik. In general we found that fancy ML techniques offer little benefit over traditional statistics for predicting social outcomes, and offered 9 hypotheses that might together explain why. msalganik.github.io/cos597E-soc555…
My coauthor @sayashk took the course and was intrigued by claims about civil war prediction that seemed to contradict this general trend. He dug deeper and found widespread methodological flaws that seem to have led to a feedback loop of overoptimism about machine learning.
To be clear, our work is a pre-print and the authors have not yet had a chance to publicly respond. We invite scrutiny of our claims; our Supplement and reproduction materials are available on the website. reproducible.cs.princeton.edu
Although ML is unlikely to produce leaps in accuracy for predicting social outcomes, small gains are often possible. If so, how do we weigh these benefits against drawbacks such as explainability? We hope to have more to say about this critical policy question. Stay tuned.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
