Thread by @johnhewtt

Does my unsupervised neural network learn syntax? In new #NAACL2019 paper with @chrmanning, our "structural probe" can show that your word representations embed entire parse trees.

paper: nlp.stanford.edu/pubs/hewitt201…
blog: nlp.stanford.edu/~johnhew/struc…
code: github.com/john-hewitt/st…
1/4

@chrmanning Key idea: Vector spaces have distance metrics (L2); trees do too (# edges between words). Vector spaces have norms (L2); rooted trees do too (# edges between word and ROOT.) Our probe finds a vector distance/norm on word representations that matches all tree distances/norms 2/4

These distances/norms reconstruct each tree, and are parametrized only by a single linear transformation. What does this mean? In BERT, ELMo, we find syntax trees approximately embedded as a global property of the transformed vector space. (But not in baselines!) 3/4

This claim, that parse trees are embedded through distances and norms on your word representation space, is a structural claim about the word representation space, like how vector offsets encode word analogies in word2vec/GloVE. We hope people have fun exploring this more! 4/4

So a lot of people have arrived here; please read
@nsaphra's excellent take on neural net probes and @nelsonfliu's comprehensive neural net probing study, both also at #naacl2019

https://twitter.com/nsaphra/status/1099978662819057664

Saphra: arxiv.org/abs/1811.00225

Liu: homes.cs.washington.edu/~nfliu/papers/…