Motivated by having seen yet another Platt’s scaler post.
Platt’s scaling and isotonic regression are ~20 years old at this point. Both of them don’t have any mathematical guarantees of validity and are outperformed by conformal prediction Venn-ABERs
#conformalprediction
Platt’s scaling and isotonic regression are ~20 years old at this point. Both of them don’t have any mathematical guarantees of validity and are outperformed by conformal prediction Venn-ABERs
#conformalprediction
VENN-ABERS is in fact a better regularised version of isotonic regression that constructors two isotonic regressions by postulating that a test object can a priori have both 0 and 1 as a label.
By doing that VENN-ABERS is able to achieve theoretical guarantees of validity (lack of bias) as the expense of multi probability prediction where the test object will have two probabilities of class 1 instead of one. p0 is a lower bound and p1 is an upper bound for class 1 prob
The actual probability of the object being of class 1 is guarantees to be within this interval. As VENN-ABERS belongs to Conformal Prediction such guarantees are standard regardless of the data distribution, the type of classifier and even of the data size.
The width of the interval (p0,p1) is a valuable information in itself as it reflects on the difficulty of classifying specific object, for decision making such interval can be easily combined into one probability (of class 1) as p = p1 / (1-p0+p1)
Unlike Platt’s scaler that is based on restrictive assumptions of underlying classifier producing sigmoid shape scores (which holds for SVM that Platt’s scaler was developed for but not for other classifiers). VENN-ABERS doesn’t make any assumptions beyond IID
And hence, being part of Conformal Prediction, offers robust mathematical guarantees.
Multiple papers have shown that VENN-ABERS scaler also often outperforms both Platt’s scaler and Isotonic regression empirically, whilst not suffering from prediction bias.
There is a Python library available where anyone can run Venn-ABERS with a few lines of Python code github.com/ptocca/VennABE…
#opensource
#opensource
There is a paper from creator of #conformalprediction Prof Vladimir Vovk proceedings.neurips.cc/paper/2015/fil…
A talk explaining the paper
As well as several tutorials about Venn ABERS on Awesome Conformal Prediction github.com/valeman/awesom…
Video tutorial
And slides cml.rhul.ac.uk/people/ptocca/…
@threadreaderapp unroll
• • •
Missing some Tweet in this thread? You can try to
force a refresh
