Sometimes validation loss < training loss. Ever wondered why? 1/5 
The most common reason is regularization (e.g., dropout), since it applies during training, but not during validation & testing. If we add the regularization loss to the validation loss, things look much different. 2/5
Oh and the training loss is measured *during* each epoch, while the validation loss is measured *after* each epoch, so on average the training loss is measured ½ an epoch earlier. If we shift it by ½ an epoch to the left (where is should be), things again look much different. 3/5
Or perhaps the val set is easier than the training set! This can happen by chance if the val set is too small, or if it wasn't properly sampled (e.g., too many easy classes). Or the train set leaked into the val set. Or you are using data augmentation during training. 4/5
Even if the val loss is close to the train loss, your model may still be overfitting. Account for the regularization loss when comparing, shift the train loss by half an epoch, and make sure the val set is large, sampled from the same distribution as train, without leaks. 5/5 🦎
