How would you interpret the situation if you train a model and see your graphs like these? ππ
#machinelearning
#machinelearning
If you just focus on the left side, it seems to make sense.
The training loss going down, the validation loss going up.
Clearly, seems to be an overfitting problem? Right?
The training loss going down, the validation loss going up.
Clearly, seems to be an overfitting problem? Right?
But the graphs on the right don't seem to make sense in terms of overfitting.
The training accuracy is high, which is fine, but why is that validation accuracy is going up if the validation loss is getting worse, shouldn't it go down too?
Is it still overfitting?
YES!
The training accuracy is high, which is fine, but why is that validation accuracy is going up if the validation loss is getting worse, shouldn't it go down too?
Is it still overfitting?
YES!
Now, why is the validation graph going upwards? Let's understand.
Loss tracks the inverse confidence of the prediction. A high loss indicates that even though the model is making correct predictions it is less sure of the predictions it makes and vice-versa.
Loss tracks the inverse confidence of the prediction. A high loss indicates that even though the model is making correct predictions it is less sure of the predictions it makes and vice-versa.
So, High Val Accuracy + High Loss Score vs High Train Accuracy + Low Train Loss suggests the model may be overfitting on train data.
Suppose there are 2 classes- dog & cat.
For our case, the correct class is a dog [1,0]
If the output of the softmax is [0.9, 0.1]
Loss would be ~0.105.
Classification-dog
If softmax output is [0.6, 0.4].
Loss would be ~0.511
Classification is same but loss has increased.
For our case, the correct class is a dog [1,0]
If the output of the softmax is [0.9, 0.1]
Loss would be ~0.105.
Classification-dog
If softmax output is [0.6, 0.4].
Loss would be ~0.511
Classification is same but loss has increased.
When both accuracy and loss are increasing, the network is starting to learn patterns only relevant for the training set and not great for generalization, some images from the validation set get predicted really wrong, with an effect amplified by the "loss asymmetry".
However, it is at the same time still learning some patterns which are useful for generalization as more and more images are being correctly classified! Interesting right!
When I first encountered this situation in my project, it left me puzzled. These were the only logical explanations I could find.
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
