Thread by @gneubig

Retrieval-based models are increasingly important in NLP/QA. But an important factor in modeling text is knowing *where* it came from. Our #ICLR2022 paper proposes retrieval-based LMs considers the "structural locality" of texts to improve retrieval: arxiv.org/abs/2110.02870 🧵↓

We demonstrate this on two example datasets: Wikipedia articles and Java code. We leveraging the article and project structure respectively to define different "locality" levels between two documents.

Our analysis shows that the distance between embeddings, used widely in retrieval tasks, is *not* capturing this locality directly, so further improvements are needed. We do this by learning a function to adjust the distance metric for each locality level in KNN language models.

Results are nice! Perplexity of the strong KNN-LM model improves, and there are good qualitative examples demonstrating why locality should be considered. We believe this is just the start! Retrieval is all over NLP now and similar methods could be applied to other models.

Thanks to @frankxu2004 for his great work on the project, together with @junxian_he, me, and @VHellendoorn!

Code is available here: github.com/frankxu2004/kn…