Thread by @DeepMind

Memory is a crucial feature of intelligence. Our new blog post overviews the use of memory in deep learning, and how modelling language may be an ideal task for developing better memory architectures:
deepmind.com/blog/article/A…

In order to understand a book, a reader must maintain memories of events and characters over many chapters. We introduce the Compressive Transformer, a neuroscience-inspired memory model that achieves state-of-the-art performance in language modelling:
arxiv.org/abs/1911.05507

as well as a benchmark dataset for book-length texts, over double the size of the Billion Word benchmark:
github.com/deepmind/pg19