Thread by @mgalle

Of the many excellent papers at #emnlp2019, this is the one I can't stop thinking about.

On NMT Search Errors and Model Errors: Cat Got Your Tongue?
by Felix Stahlberg & Bill Byrne
aclweb.org/anthology/D19-…

Our current NMT models might be "wrong"🚫

👇thread 👇 1/7

2/7. One relatively well known open dirty secrets of NMT was that if you increase the beam width at decoding, your BLEU sometime decreases

3/7. This shouldn't happen. With ♾-width, you should get the "best" translation, and (n+1)-width > n-width

4/7. ♾-width is exactly what Felix & Bill do, using a neat trick that allows them to trim huge chunks of the search space

5/7. Once you have this in place, you can now compute the best translation sequence of your model.

The surprise is that in 51.8% this is the empty sequence.

In half of the cases, your model thinks that the best output translation is nothing.

6/7

This is not a case of "Even a fool, when he keeps silent, is considered wise".

Longer sequence have lower probability, so while a single EOS token is very unlikely, it is more likely than producing anything long.

7/7
This is very disturbing, as we are all playing with models whose best behavior is horrible.

It also points to two unsatisfactory practices:
- inference is done differently than training (beam search)
- length of translations is handled with EOS. Is this the best way?