Thread by @rohanpaul_ai

A thread on AUC Score (Area under the ROC Curve) Interpretation in #DataScience #MachineLearning

1/16

#DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics #pythoncode #AI #ArtificialIntelligence #TensorFlow #PyTorch #Pandas

2/16

"roc_auc_score" is defined as the area under the ROC curve, which is the curve having False Positive Rate on the x-axis and True Positive Rate on the y-axis at all classification thresholds.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python

3/16

AUC ranges in value from 0 to 1.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics #pythoncode #AI #ArtificialIntelligence #TensorFlow #PyTorch #Pandas #Stat #dataviz #learning

4/16

AUC is equivalent to the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics

5/16

A model whose predictions are 100% wrong has an AUC of 0. and one whose predictions are 100% correct has an AUC of 1

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics #pythoncode #AI

6/16

In other words - roc_auc_score coincides with “the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one”.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics

7/16

AUC is scale-invariant. It measures how well predictions are ranked, rather than their absolute values.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics #pythoncode #AI

8/16

AUC is classification-threshold-invariant. It measures the quality of the model's predictions irrespective of what classification threshold is chosen.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data

9/16

Some caveats usefulness of AUC in certain use cases are as follows:

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data #DataAnalytics #pythoncode #AI #ArtificialIntelligence #TensorFlow #PyTorch #Pandas

10/16

Scale invariance is not always desirable. e.g, sometimes we really do need well calibrated probability outputs, and AUC won’t tell us about that.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python #DataScientist #Statistics #programming #Math #Data

11/16

Classification-threshold invariance is not always desirable, in cases where there are wide disparities in the cost of false negatives vs. false positives, it may be critical to minimize one type of classification error.

#DataScience #MachineLearning #100DaysOfMLCode

12/16

e.g, in email spam detection, you likely want to prioritize minimizing false positives (i.e. an Email is NOT spam, but its positively determined as a spam and hence moved to span folder).

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Statistics

13/16

Even if that results in a significant increase of false negatives. (An email is indeed spam, but model determines it to be negative, i.e. Not-Spam). AUC isn't a useful metric for this type of optimization.

#DataScience #MachineLearning #Statistics #DeepLearning

14/16

How to use the AUC ROC curve for the multi-class model ?

In a multi-class model, we can plot the N number of AUC ROC Curves for N number classes using the One vs Rest methodology.

#DataScience #MachineLearning #100DaysOfMLCode #Python #DataScientist #Statistics

15/16

“One vs Rest” is a method to evaluate multiclass models by comparing each class against all the others at the same time. Here we take one class and consider it as our “positive” class, while all the others (the rest) are considered as the “negative” class.

#DataScience

16/16

e.s. if you have three classes named X, Y, and Z, you will have one ROC for X classified against Y and Z, another ROC for Y classified against X and Z, and the third one of Z classified against Y and X.

#DataScience #MachineLearning #DeepLearning #100DaysOfMLCode #Python