Thread by @NicoWolf16

A translation of my first thread for the general public out there. I will talk about how to correctly, yet efficiently model the uncertainty on predictions (for example in machine learning). (1/n)

#statistics #DataScience #machinelearning #conformalprediction

When I started as a PhD I was convinced of two things:
1) Modelling uncertainty is hard, and
2) The only viable approach is the Bayesian one.

This idea is so strongly ingrained in the statistical literature and data science community that it must be true, right? (2/n)

The answer is no and luckily I quickly learned of a great alternative. The idea behind "Conformal prediction" is as simple as possible: You calculate the errors on a holdout dataset and choose, for example, the 90% quantile. (3/n)

If you include in your prediction intervals (the confidence intervals for new predictions) all those values that deviate at most as much from your prediction as the chosen quantile, your intervals will be correct in 90% of the cases. (4/n)

In sharp contrast to most other methods, the intervals coming from this approach will be correct in a statistical sense (everybody will be happy) and in contrast to Bayesian models, you do not need a decent guess of the underlying distribution. (5/n)

The only thing you need is that the order of the data points does not matter (this holds for almost all data sets without a time aspect). Moreover, the framework is so much more efficient and straightforward than the numerical nightmare that is Bayesian modelling. (6/n)

If you would like to know more about conformal prediction and how it behaves when combined with existing models, read our paper arxiv.org/abs/2107.00363 or follow @predict_addict with his awesome library. (7/n, n=7)