Thread by @ChristophMolnar

Machine learning sucks at uncertainty quantification.

But there is a solution that almost sounds too good to be true:

conformal prediction

• works for any black box model
• requires few lines of code
• is fast
• comes with statistical guarantees

A thread 🧵

Conformal prediction is a method for uncertainty quantification of machine learning models.

The method takes a heuristic uncertainty score and turns it into a rigorous one.

What confused me first about conformal prediction: It's not 1 algorithm, but a general recipe:

• Split data: training and calibration
• Train model
• Calculate "heuristic" scores on calibration data
• Calibrate score
• Use calibrated scoring rule on new data

The recipe enables many use cases:

• Prediction sets in multi-class problems
• Calibrate classification scores so they can be interpreted as probabilities
• Fix coverage of quantile regression
• Conformal predictive distributions for regression
• ...

How to get started with conformal prediction:

• Tutorial paper: arxiv.org/abs/2107.07511
• Awesome list of conformal prediction approaches: github.com/valeman/awesom…
• scikit-ready Python package: github.com/scikit-learn-c…
• R package: github.com/ryantibs/confo…

Subscribe to my newsletter Mindful Modeler to learn more about conformal prediction and other techniques to make machine learning better.

mindfulmodeler.substack.com