“Our sense of progress largely rests on a small number of standard benchmarks such as CIFAR-10, ImageNet, or MuJoCo. This raises a crucial question: How reliable are our current measures of progress in machine learning?” 🔥
I would love to see a similar study for text and translation, and see how well network architectures that are “hyper” optimized to death for PTB, wikitext, enwik8, WMT’14 EN-FR, EN-DE, etc. transfer to “new” test sets from same distribution, or even perhaps to other text domains.
If we see similar results for PTB, then this is actually quite good news for deep learning research community, where the typical process of hyper optimizing for performance on a smallish dataset *does* lead to discovery of new methods that generalize better.
I would also be interested to see how well AutoAugment performs for the new test set.
