Thread by @paraschopra

1/ A thread on MACHINE LEARNING and why it's INEFFICIENT.

Let's start with a simple idea. Imagine we are asked to predict and our simulation overlords feed images of the following type to "learn" from.

What would you learn from white noise? Will you be able to extract patterns?

2/ In all likelihood, the alien world will be incomprehensible to us (as it would have a very different nature than our world).

Hence, whatever data is fed to us will seem RANDOM (as, in our world, we’d have never come across it before).

3/ Having zero background knowledge about the alien world, you’d not assume anything and hence will require lots of data points tease out any correlations present in the dataset.

4/ For example, maybe, when aliens mapped their data to our 2D visual field, the real determiner of whether it corresponds to a specific label or not is a combination of the 5th, 99th, and 213th pixel.

Or maybe they're messing with us and feeding us truly random data.

5/ The difficulty in finding patterns in the alien dataset is that, because we know nothing about the alien world, we’d need to have an infinite number of such hypotheses on what constitutes an input -> output mapping.

6/ This mapping could literally be anything and a few examples of data won’t suffice.

To "LEARN" from alien data set, we'd require tons of data because it will "LOOK" to us as completely random

7/ Key insight -> FOR MACHINES, OURS IS AN ALIEN WORLD

8/ Imagine what it would like to be a deep neural network trying to label images of handwritten digits.

The MNIST dataset is a famous dataset in ML and modern algorithms have achieved a “near human accuracy” on predicting which image corresponds to which digit.

9/ But getting to that “near human accuracy” took SIXTY THOUSAND examples (the size of db) fed into the algorithm. A human child can start recognizing digits with only a few examples.

Why is the ML algo so inefficient as compared to a human child?

10/ The key to understanding the reason behind this inefficiency is to appreciate that what looks like a “1” to us looks like white noise to an algorithm.

11/ Just like an alien data set will likely look random and incomprehensible to us.

To an algorithm, the following two images are exactly the same because it knows nothing about our world.

12/ But humans learn handwritten digits only using a few examples (and not tens of thousands like ML algos). Why?

The secret to quick learning in humans: EVOLUTION

13/ Yes, Evolution really does explain everything, including why humans are able to learn generic patterns using only a few examples.

I wrote about evolution in my last essay: invertedpassion.com/evolution-expl…

14/ Machine learning algorithms are written from the scratch, while we evolved.

Our existence is a result of several billion years of evolution where organisms who “knew” more about the world survived better than the ones who had no clue.

15/ For example, a deer who didn’t “know” that a moving predator will soon reach close to it died pretty soon. Similarly, a deer who didn’t “know” that broken tree branches fall down also died pretty soon.

16/ On the other hand, the deer who had the “hunch” of running away from moving objects learned that moving things collide and hence survived.

17/ In this way, the process of EVOLUTION ENCODES INFORMATION about the world in an organism at a very DEEP LEVEL.

That is why we do not have to “think” and solve for Newton’s law of motion to know that a flying spear can hit us.

18/ In case of handwritten digits, it is our background knowledge that helps make learning from a few examples possible.

The first piece of knowledge is about HOW HUMAN HANDS WORK. While writing, rather than pushing the pen in random directions, we always make strokes.

19/ It’s a fact that most (all?) written languages have characters which require stroke-like patterns.

If you know that fact about human hands, it’s far easier to guess what strokes constitute what digits, rather than what exact sequence of pixels constitute what digits.

20/ On the contrary, an ML algo DOESN'T know anything about human hands.

For an algo with 0 background knowledge, first, it has to learn that we use strokes, not pixels. Then, it has to learn that we use few strokes rather than a million strokes to write digits.

21/ So it’s no surprise that machine learning algorithm require such a large training data set to make any sense of our world.

22/ This insight explains why machine learning algorithms are MUCH BETTER than humans at tasks that are more recent in our history (chess, detecting cancer, etc.)

These problems have a structure that’s very different than what’s found in nature.

23/ The sample space of valid chess moves is mathematical and has no analog in the world where humans evolved.

24/ Unlike recognizing handwritten digits where learning is “automatic”, humans have to master chess through years of practice.

So it’s no wonder that a machine learning algorithm can practice a lot more and get much better than humans at chess. theguardian.com/technology/201…

25/ I love this lecture by @tegmark where he hints at what deep neural networks might be doing.

They might actually be ENCODING STRUCTURE of OUR PHYSICAL UNIVERSE, which is exactly what evolution does in organisms.

26/ Next time you wonder why a ML algo can’t do something that occurs to you as “common sense” (such as telling jazz/blues apart, or laughing at a joke), know that evolution has taken billions of years to embed in you a “gut knowledge” about these things while ML starts from zero

27/ That's all folks.

All my tweetstorms are derived from my essays. This one is from: invertedpassion.com/inefficiency-m… <- SUBSCRIBE to get email updates.

RT if you have any ML/AI/Evolution fans as your followers :)