Neat negative result spotted at #ACL2021:
I've seen a number of efforts that try to use MNLI models to do other classification tasks by checking whether the input entails statements like 'this is a negative review'. (1/...)
I've seen a number of efforts that try to use MNLI models to do other classification tasks by checking whether the input entails statements like 'this is a negative review'. (1/...)
This never really made sense. The data collection process behind SNLI/MNLI was meant to capture the relationship between two things that the same speaker could have said in the same situation.
That means strings like 'the text' or 'the premise' or 'the author' are rare in MNLI, and when they appear, they refer to something _that was referred to in the premise_, not to the premise itself or to its author.
So, examples like the one in the screenshot seem broken—that kind of hypothesis doesn't make sense in the context of most presises, and we should expect models to be confused, and to behave erratically.
And they do! Tingting Ma et al. from MSR find that when using this kind of prompting, you'll actually do _better_ if you use a next-sentence-prediction model rather than an MNLI model, ...
... suggesting that the limited successes we've seen here have to do with a vague ability for BERT-style models to recognize topic/style similarity, and that the specific abilities that a model learns from MNLI aren't really helping. The world makes sense!
• • •
Missing some Tweet in this thread? You can try to
force a refresh
