Thread by @Eric_Wallace_

Most NLP models treat numbers (e.g., “91”) in the same way as other tokens, i.e., they embed them as vectors. Is this a good representation for downstream numerical tasks such as DROP, math questions, etc?

Yes! Pre-trained vectors (BERT, GloVe, ELMo) know numbers.[1 / 6]

We begin by testing QA models on questions that evaluate numerical reasoning (e.g., sorting, comparing, or summing numbers), taken from the DROP dataset. Standard models excel on these types of questions! [2 / 6]

To understand how models can capture numerical reasoning, we probe their token embeddings (e.g., BERT, GloVe) using list maximum, number decoding, and addition tasks. [3 / 6]

Pre-trained embeddings capture numeracy: number magnitude/order is present, even for numbers in the thousands. Character-level methods are especially strong (ELMo excels while BERT is mediocre), and CNNs are a great prior for numeracy. [4 / 6]

We do find one failure mode---the probing models struggle to extrapolate to numbers outside the training range. See in-domain test set predictions (blue) versus out-of-domain (red) in the first tweet. We attribute this problem to neural models themselves. [5 / 6]

See all of this and much more in:

Do NLP Models Know Numbers? Probing Numeracy in Embeddings by @Eric_Wallace_ @yizhongwyz, Sujian Li, @sameer_ @nlpmattg

#emnlp2019

Paper: arxiv.org/abs/1909.07940