Thread by @Jeande_d

The machine learning research community is very and very vibrant.

Here is what I mean...🧵🧵

In 1958, Frank Rosenblatt invented a perceptron, a very simple algorithm that would later turn out to be the core and origin of to days intelligent machines.

In essence, the perceptron is a simple binary classifier that can determine whether or not a given input belongs to a specific class.

Here is the algorithm of perceptron:

Rosenblatt's intent was not to develop a perceptron as an algorithm or software, but rather a machine.

The perceptron was implemented in hardware that got the name of Mark I Perceptron.

The weights were encoded in potentiometers, and weight updates were done by electric motors.

Here is Mark I Perceptron.

A perceptron or a simple artificial neural network architecture led to multilayer perceptrons(MLP) and later inspired other types of neural networks such as convolutional neural networks and recurrent neural networks.

Let's shift a little bit to convolutional neural networks.

Convolutional neural networks are the type of neural nets that have succeeded in image recognition tasks.

They are also used in text classification and time series analysis, but their major applications lie in visual recognition tasks.

One of the first Convnets architecture is LeNet-5 which was introduced by @ylecun.

LeNet-5 was designed for handwritten and machine-printed character recognition.

Here is a 1998 paper that used LeNet-5 in text recognition. The first paper that used LeNet-5 was in 1989.

vision.stanford.edu/cs598_spring07….

LeNet-5 inspired other powerful Convnets architectures such as AlexNet that won the Imagenet challenge of 2012.

In the years that followed, nearly all architectures that won the Imagenet challenge used Convnets.

Here is AlexNet paper

papers.nips.cc/paper/2012/fil…

Most of the architectures that won the preceding challenges were a modification of previous architecture.

Example: ZFNet which was a modification of AlexNet won the Imagenet challenge in 2013.

arxiv.org/abs/1311.2901

Other Convnets architectures that followed include GoogleLeNet by @ChrSzegedy(won imagenet 2014 and inspired Inception v-3 and v-4), VGG or Visual Geometry Group (that won Imagenet runner-up 2014), and ResNet that won the 2015 challenge.

There are other Convnets architectures that followed but most of them were modifications of previous architectures.

For example, Xception or Xtreme Inception by @fchollet was a version of GoogLeNet that used separable convolution layers instead of inception modules.

Now turning to recurrent networks.

Recurrent neural networks were created to handle sequential data such as texts, audio, and time series.

The first simple RNN cells failed to handle long-term sequences due to the problem of vanishing gradients, and Long Short Term Memories(LSTMs) were introduced to overcome that.

Gated Recurrent Units(GRUs) which is a simple version of LTSMs were later introduced too.

More about vanishing gradients problems

https://twitter.com/Jeande_d/status/1436277279697539074?s=20

LSTMs are GRUs are all recurrent networks and are widely used in sequential tasks today.

BUT later, guess what? You didn't need recurrence or convolutions. You only need attention.

That was the slogan of Transformers. Attention is All You Need!

That was in 2017.

Transformers are solely based on attention mechanisms and they don't use any recurrence or convolutions.

arxiv.org/abs/1706.03762

They are now used in things like machine translations and other language tasks.

Real example: @lmoroney used transformers for his exciting project

https://twitter.com/lmoroney/status/1441410875458396163?s=20

Transformers have also been used in visual tasks.

See this.

arxiv.org/abs/2105.07581

It also turns out that you don't need attention or convolutions.

MLP-Mixer is one of the papers that claimed that none of Convnets or vision transformers is needed.

Multilayer perceptrons are enough...That was in May this year.

arxiv.org/abs/2105.01601…

And in the same month, another similar research claimed the same thing - YOU DON'T NEED ATTENTION. PAY ATTENTION TO MLPs.

PERCEPTRONS ARE ALL YOU NEED.

arxiv.org/abs/2105.08050

And this week, guess what? You don't need anything other than patches.

This was shocking news to the whole ML community.

See this from @karpathy

https://twitter.com/karpathy/status/1445915220644229124?s=20

And here is ResNet again...

See this from @tunguz

https://twitter.com/tunguz/status/1445773541274632197?s=20

In the beginning, it was perceptrons. And to day, ML research is still revolving around perceptrons.

It's good to be alive watching all of this incredible news!

Thanks for reading.

If you found any of this thing helpful, follow @jeande_d and share this thread with your friends.

For an in-depth understanding of neural networks architectures,

Here is a whole thread about them 👇

https://twitter.com/Jeande_d/status/1436666068559437827?s=20