Transformers pioneered the principle of attention mechanisms to access past information.

However, most Transformer models discard older memories to prioritize more recent activations.

@DeepMind's Compressive Transformer tackles that problem.
1/4
The Compressive Transformer tries to imitate the process of consolidating memories.

Under that approach, previous activations are compacted into a "compressed memory" that can be used in long-range tasks.
2/4
Compressive Transformer was evaluated against state-of-the-art memory models using WikiText-103  and  Enwik8. 

In both cases, it showed significant improvements over more established models both in memory and efficiency.
3/4
Thanks for being with us!

You can find additional info on Compressive Transformers in the original @DeepMind's paper: arxiv.org/abs/1911.05507

We've discussed the concept of attention used by transformer models here:
4/4

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with TheSequence

TheSequence Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @TheSequenceAI

5 Nov
🔥2 New Super Models to Handle Any Type of Dataset

We build models optimized for a specific type of dataset like:
- text
- audio
- computer vision
- etc.

Is it possible to create a general model? @DeepMind unveils the answer⬇️
1/5
Recently, DeepMind published two papers about general-purpose architectures that can process different types of input datasets.

1) Perceiver supports any kind of input
2) Perceiver IO supports any kind of output

More⬇️
Perceivers can handle new types of data with only minimal modifications.

They process inputs using domain-agnostic Transformer-style attention.

Perceiver IO matches a Transformer-based BERT baseline on the GLUE language benchmark.
3/5
Read 5 tweets
4 Nov
3 reasons why you need to outsource data labeling.

1) Teams want to invest time in ML models, not in data-centric operations

2) You care about the amount and quality of labeled data

3) The entire data annotation process involves a lot of steps
⬇️
1) The majority of the time invested in an AI project is allotted to data-centric operations

Data labeling methods keep being increasingly important to the success of ML solutions.

The process can be overwhelming. Especially for startups and small companies.
2) You care about the amount and quality of labeled data

The success of supervised learning depends extensively on these parameters.

Labels guide the ML model in the right direction such that it can classify unseen samples accurately.
Read 5 tweets
30 Oct
.@OpenAI ImageGPT is one of the first transformer architectures applied to computer vision scenarios.👇
In language, unsupervised learning algorithms that rely on word prediction (like GPT-2 and BERT) are extremely successful.

One possible reason for this success is that instances of downstream language tasks appear naturally in the text.
2/4
In contrast, sequences of pixels do not clearly contain labels for the images they belong to.

However, OpenAI believes that sufficiently large transformer models:
- could be applied to 2D image analysis
- learn strong representations of a dataset
3/4
Read 4 tweets
29 Oct
Forecasting high-dimensional time series plays a crucial role in many applications like:
- demand forecasting
- financial predictions

You can use @AmazonScience's DeepGLO for these problems.⬇️
The challenge with multi-dimensional time-series datasets is a serious one.

1) Traditional methods (like ARIMA) can't scale to large datasets with millions of time series.

2) Deep neural networks have been proven to handle scalability more effectively. BUT⬇️
BUT many deep neural nets:

- only forecast values from the same dimension
- require different time series to be normalized on a single scale

DeepGLO addresses these challenges.
3/6
Read 6 tweets
29 Oct
There are a handful of frameworks to implement basic NLP.

And what about implementing models like BERT or GPT-3? A framework that does not require monumental development efforts.

@allen_ai created one for you. It's AllenNLP.⬇️
AllenNLP provides a simple & modular programming model for:

1. Applying advanced deep learning techniques to NLP research
2. Streamlining the creation of NLP experiments
3. Abstracting the core building blocks of NLP models

2/5
Portfolio of NLP tasks under AllenNLP:

- Text Generation
- Language Modeling
- Multiple Choice
- Pair Classification
- Structured Prediction
- Sequence Tagging
- Text + vision
3/5
Read 5 tweets
27 Oct
3 big AI industry insights🔥

1) Companies are big spenders on AI but lack confidence
2) AI is a cloud-native world
3) Budgets are growing, despite challenges

Fascinating details👀⬇️
1) Big spenders, but a lack of confidence

- 38% of companies have a budget of more than $1M per year for AI infrastructure alone!

- However, for 77% of companies, less than half of models make it to production 38% of companies have a budget of more than $1M per year for
3) AI is a cloud-native world

- 81% of companies use containers and cloud technologies for their AI workloads

- Nearly 1/2 of them are using @kubernetesio

=> AI is a leader in cloud-native adoption 81% of companies use containers and cloud technologies for t
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(