Thread by @DellAnnaLuca

GETTING AI/MLs TO UNDERSTAND CONTEXT
(thread)

Why do computers need to be shown 1000 coffee cups to understand what a cup is, whereas a human child can understand the concept after being shown a single cup?

Because human brains can do unsupervised contextualization.

1/N

2/ When shown a picture, Machine Learning recognizes "features".

A human recognizes "features in a context".

Wait - aren't MLs able to recognize contexts as well?

Perhaps - but when they do, they treat them as features.

Let me explain.

3/ First, a caveat. When I talk about AI/MLs, I am looking at the most common frameworks out there. I do not exclude the existence of frameworks which implement the tools I propose in the 2nd part of this thread.

Also, this is a short thread. Papers here luca-dellanna.com/research

4/ Most ML frameworks treat context as a feature it recognized in the image, to be weighted together with features to determine meaning.

For example, in the pic below, an ML would treat "kitchen" and "microwave" as two features it recognized.

Instead, the human brain…

5/ Instead, the human brain would see "a microwave in the kitchen" as a single contextualized feature: a single data point, a single concept.

I explain the neurological reasons for this statement at page #2 here: luca-dellanna.com/wp-content/upl…, but let's see why the distinction matters

6/ In the Cortex, as perception flows upwards in a hierarchy of brain regions, two operations are applied one after the other, over and over: pattern recognition and pattern contextualization. I also call them compression and expansion, respectively.

Let's see them one by one.

7/ In pattern recognition, a group of low-level contextualized patterns are grouped together and recognized together as a single pattern deprived of context.

For example: a handle, a cylinder and a concavity are recognized as an empty coffee mug.

8/ In pattern contextualization, the pattern recognized at the previous step is assigned tentative contexts, based on what *the rest* of the brain perceives as being the context.

For example…

9/ For example, if I'm at my friend Alan's house, the empty coffee mug is assigned *tentative* contexts and becomes "Alan's cup", "My cup", "A dirty cup", "A clean cup".

Some of these contexts are contradictory, but it's okay. The current brain region lacks info to discriminate.

10/ As the current brain region (the region the perceptual information is currently passing through) does not have information outside its scope, it cannot discriminate between possibilities, so treats them all as possible.

Now, the magic is in the following step.

11/ As the information {"My cup", "Alan's cup", etc} flows to the next brain region above in the hierarchy, that region applies again an operation of pattern recognition, this time grouping higher level features.

12/ The inputs come not only from the previous region {"My cup", "Alan's cup", "A dirty cup", "A clean cup"} but also from other regions {"In the sink"} at the same level of the "pyramid of perception"

The output of pattern recognition is "Alan's dirty cup".

13/ Notice a subtlety: what was context at a lower level of perception ("We're at Alan's house") now becomes part of the pattern ("Alan's dirty cup").

As perception moves up the hierarchy, it loses detail about features and it acquires meaning.

14/ In my paper luca-dellanna.com/wp-content/upl… I explain how alternating pattern contextualizations ("expansions") & pattern recognitions ("compressions") allows for the emergence of meaning (content transforms from "feature detail" to "features in context" aka meaningful features).

15/ In that paper (luca-dellanna.com/wp-content/upl…), I also propose some (patent pending) techniques to reproduce this processes in ML systems.

16/ Some FAQs:

Q: What if context is an emergent property of a deeply connected network being trained for a very long time?

17/ A: Our brain uses two very different representations for features and contexts.

Features are represented by patterns of activated neural columns; Contexts, by patterns of activated neurons.

This is lost in MLs which considers both as "data points" or "nodes" equally encoded

17/ Second: it takes years for human children to understand context, but that is rather due to a hierarchy of patterns (we need to learn to recognize lines to see letters, letters to see words, etc.; psyarxiv.com/erxp4). However, an adult can learn a new context in seconds.

18/ I can show you a single 5-seconds video of how a tool you've never seen works and you'd understand most of it (example below). Imagine a current-generation ML doing that.

https://twitter.com/ThingsWork/status/1174036628068716544?s=20

19/ Third, for context to emerge, an ML system must have a separate system to encode it from the one used for patterns.

The human 🧠 uses patterns of column activations to represent the former & patterns of neuron activations for the latter

Current ML systems do not have that

20/ I really recommend reading this paper as it has charts and it explains much better than I could ever be in a characters-limited Twitter thread the architecture used by the human brain to extract meaning from perception & how it could be applied to ML luca-dellanna.com/wp-content/upl…