Thread by @vivwylai

Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling by @XiaochuangHan and @jacobeisenstein
Link: aclweb.org/anthology/D19-…
Code: github.com/xhan77/AdaptaB…
#emnlp2019

Since BERT is trained on Wikipedia, Han and Eisenstein are interested in scenarios wheee labeled data exists only for canonical source domain, and target domain is distinct from both pretraining and labeled source domain.
#emnlp2019

One of the tasks is part of speech tagging for Early Modern English. Given labeled data in 20th century English, they solve for unlabeled data in 15-17th century English.
#emnlp2019

They do domain adaption fine-tuning in 2 steps. 1) extra masked language modeling in both source and target domains; 2) fine-tune with labeling objective.
#emnlp2019

They tested their method using BERT and call it AdaptaBERT. On the task on Early Modern English, AdaptaBERT achieves the best performance, and this is due to significant accuracy improvement in target domain out of vocab words. (end)
#emnlp2019