Discover and read the best of Twitter Threads about #conformalprediction
Most recents (7)
Awesome Conformal Prediction - The most comprehensive professionally curated resource on Conformal Prediction by @predict_addict github.com/valeman/awesom…
#2 Conformal Prediction by @ml_angelopoulos github.com/aangelopoulos/…
Conformal prediction for #timeseries #forecasting is the hottest 🔥🔥🔥🔥 🚀🚀🚀🚀 area of research and applications right now.
The whole story started in summer 2021 when researchers from Georgia Tech published the first ever paper applying conformal prediction to time series.
The whole story started in summer 2021 when researchers from Georgia Tech published the first ever paper applying conformal prediction to time series.
and presenting it at the first event at major conference totally dedicated to conformal prediction (ICML DFUQ 2021 organised by @ml_angelopoulos and @stats_stephen
Roll forward just 1.5 yeast - there are multiple papers published on the subject, several #opensoure libraries like MAPIE implemented the original first ever paper from Georgia Tech (method called EnbPI).
Machine learning predictive uncertainty estimates are often unreliable—data shift makes things worse!
How can you audit the uncertainty of an ML prediction, even with biased data?
A 🧵 w/ @DrewPrinster on the JAWS approach in #NeurIPS2022 paper w/ fab @anqi_liu33 @DrewPrinster
How can you audit the uncertainty of an ML prediction, even with biased data?
A 🧵 w/ @DrewPrinster on the JAWS approach in #NeurIPS2022 paper w/ fab @anqi_liu33 @DrewPrinster
Why generate uncertainty intervals and enable real time audits?
Build user trust arxiv.org/pdf/1805.11783… proceedings.mlr.press/v89/schulam19a…
In decision support apps, reduce false alerts pubmed.ncbi.nlm.nih.gov/28841550/
Enable safety assessment inhttps://www.nejm.org/doi/full/10.1056/NEJMc2104626
Build user trust arxiv.org/pdf/1805.11783… proceedings.mlr.press/v89/schulam19a…
In decision support apps, reduce false alerts pubmed.ncbi.nlm.nih.gov/28841550/
Enable safety assessment inhttps://www.nejm.org/doi/full/10.1056/NEJMc2104626
Background: #conformalprediction is becoming popular for predictive interval generation with a coverage guarantee
Coverage: Predictive interval contains true label with high probability (i.e., predictive confidence intervals are valid)
Assumption: Exchangeable (or, IID) data
Coverage: Predictive interval contains true label with high probability (i.e., predictive confidence intervals are valid)
Assumption: Exchangeable (or, IID) data
A translation of my first thread for the general public out there. I will talk about how to correctly, yet efficiently model the uncertainty on predictions (for example in machine learning). (1/n)
#statistics #DataScience #machinelearning #conformalprediction
#statistics #DataScience #machinelearning #conformalprediction
When I started as a PhD I was convinced of two things:
1) Modelling uncertainty is hard, and
2) The only viable approach is the Bayesian one.
This idea is so strongly ingrained in the statistical literature and data science community that it must be true, right? (2/n)
1) Modelling uncertainty is hard, and
2) The only viable approach is the Bayesian one.
This idea is so strongly ingrained in the statistical literature and data science community that it must be true, right? (2/n)
The answer is no and luckily I quickly learned of a great alternative. The idea behind "Conformal prediction" is as simple as possible: You calculate the errors on a holdout dataset and choose, for example, the 90% quantile. (3/n)
Mijn eerste draadje (zoals men dit blijkbaar noemt) gaat over het correct, maar eenvoudig modelleren van de onzekerheid op voorspellingen. (1/n)
#Statistics #DataScience #machinelearning #conformalprediction
#Statistics #DataScience #machinelearning #conformalprediction
Toen ik begon als PhD was ik van twee dingen overtuigd:
1) Onzekerheid op voorspellingen modelleren is niet eenvoudig.
2) Dit kan enkel op een Bayesiaanse manier.
Dit idee is zo sterk verspreid binnen de statistiek en datawetenschappen dat het wel waar moet zijn. Of niet? (2/n)
1) Onzekerheid op voorspellingen modelleren is niet eenvoudig.
2) Dit kan enkel op een Bayesiaanse manier.
Dit idee is zo sterk verspreid binnen de statistiek en datawetenschappen dat het wel waar moet zijn. Of niet? (2/n)
Gelukkig leerde ik al vrij snel een alternatief kennen. Het idee achter "Conformal prediction" is zo simpel als het maar kan zijn: je berekent de fouten op wat validatiedata en kiest bijvoorbeeld het 90%-kwantiel. (3/n)
Motivated by having seen yet another Platt’s scaler post.
Platt’s scaling and isotonic regression are ~20 years old at this point. Both of them don’t have any mathematical guarantees of validity and are outperformed by conformal prediction Venn-ABERs
#conformalprediction
Platt’s scaling and isotonic regression are ~20 years old at this point. Both of them don’t have any mathematical guarantees of validity and are outperformed by conformal prediction Venn-ABERs
#conformalprediction
VENN-ABERS is in fact a better regularised version of isotonic regression that constructors two isotonic regressions by postulating that a test object can a priori have both 0 and 1 as a label.
By doing that VENN-ABERS is able to achieve theoretical guarantees of validity (lack of bias) as the expense of multi probability prediction where the test object will have two probabilities of class 1 instead of one. p0 is a lower bound and p1 is an upper bound for class 1 prob
Understanding data quality is crucial for reliable ML. In our #ICML2022 paper, @NabeelSeedat01, @JonathanICrabbe & @MihaelaVDS present a Data-Centric framework for the understudied problem of identifying incongruous examples of in-distribution data.
🧵1/10
🧵1/10
TLDR.
*Do you want to know which examples will be reliably predicted, independent of the downstream predictive model?
* Do you want to get insights into your data to understand possible limitations?
If so, Data-SUITE our new #DataCentricAI framework is for you!
2/10
*Do you want to know which examples will be reliably predicted, independent of the downstream predictive model?
* Do you want to get insights into your data to understand possible limitations?
If so, Data-SUITE our new #DataCentricAI framework is for you!
2/10
There has been a significant focus on out-of-distribution data (OOD) for reliable ML.
However, in Data-SUITE we tackle an equally important yet understudied problem.
How do we assess In-Distribution data, with feature space heterogeneity?
3/10
However, in Data-SUITE we tackle an equally important yet understudied problem.
How do we assess In-Distribution data, with feature space heterogeneity?
3/10
