,
4 tweets,
2 min read
Read on Twitter
Niven & Kao's upcoming #acl2019nlp paper "Probing Neural Network Comprehension of Natural Language Arguments" asks exactly the right question of unreasonable performance: "what has BERT learned about argument comprehension?"
Preprint:
arxiv.org/abs/1907.07355
/1
Preprint:
arxiv.org/abs/1907.07355
/1
They show, with careful experiments, that in fact, “BERT has learned nothing about argument comprehension.” Rather: “As our learners get stronger, controlling for spurious statistics becomes more important in order to have confidence in their apparent performance.” /2
This kind of careful work, featuring careful attention to the data, is exactly what #NLProc needs more of! /3
(This paper concerns the ARCT dataset, which, though Niven & Kao don't say so, appears to involve English. This kind of result, that BERT is attending to spurious statistical artifacts, is one that I would expect to hold for any language with sufficient data to train BERT.) /fin
