Thread by @ml_angelopoulos

I’m thrilled to announce Conformal Risk Control: a way to bound quantities other than coverage with conformal prediction.

arxiv.org/abs/2208.02814

Check out the worked examples in CV and NLP!

The best part is: it’s exactly the same algorithm as split conformal prediction🤯🧵1/5

Conformal prediction bounds the miscoverage:
P(Y notin set)<=α.

Our paper shows that conformal prediction actually works to bound any risk,
E[L(set,Y)]<=α,
for some loss L that shrinks as the set grows.

L(set,Y)=I(Y notin set) reduces to conformal, but you can do more... 2/5

In multilabel classification, we can take L to be the FNP, and thereby bound the fraction of missed classes.

In image segmentation, we can do the same thing and bound the fraction of missed tumor pixels.

In question-answering, we can bound the F1-score.

All with conformal🔥3/5

The same algorithm we all know and love goes far beyond prediction intervals.

For risks, it has exactly the same coverage guarantee, and a similar O(1/n) upper-bound.

We were shocked, even after working on risk control for two years. The simplicity is 🍰 4/5

This is the result of a huge collaboration with @stats_stephen, @adamjfisch, @lihua_lei_stat, @TalSchuster.

It's a short paper --- the theory section is about 2 pages long --- hopefully you find it as interesting and easy-to-use as we did.😁

Code: github.com/aangelopoulos/…

5/5