The news headlines *undersold* this paper. Widely-used machine learning tool for sepsis prediction found to have an AUC of 0.63 (!), adds little to existing clinical practice. Misses two thirds of sepsis cases, overwhelms physicians with false alerts. jamanetwork.com/journals/jamai…
This adds to the growing body of evidence that machine learning isn't good at true prediction tasks as opposed to "prediction" tasks like image classification that are actually perception tasks.
https://twitter.com/random_walker/status/1196870349574623232
Worse, in prediction tasks it's extremely easy to be overoptimistic about accuracy through careless problem framing. The sepsis paper found that the measured AUC is highly sensitive to how early the prediction is made—it can be accurate, or clinically useful, but not both.
Fortunately in medicine there's a well established set of methods for testing if something works, so the BS eventually gets called out. I worry much more about domains like résumé screening where the accuracy claims of ML-based prediction tools are never effectively challenged.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
