Why are models that curve fit not compositional models? medium.com/intuitionmachi…
Why are neural networks unable to nail arithmetic or multiplication? That is, you might be able to ask GPT-3 what 5 plus 7 equals, but you can't calculate 59 + 77 (trust me, it can't). Why is that?
This is because neural networks are unable to formulate compositional models of reality. Would a caveman be able to invent arithmetic or multiplication? I seriously doubt it, it requires a gifted human individual to invent these from scratch.
But computers can easily compute 59 + 77. That's because we've encoded the algorithms (i.e. the step by step procedures) of how to compute the results. You see, knowledge is not a collection of facts, it's a collection of skills.
The curious thing about skills is that they are composable. Just add one skill after another as the most simplest case. In the more complex case, you just add conditional expressions and loops.
Programmers are very good at instructing computers as to what to precisely do. What we don't have yet are algorithms that are good at instructing computers as to how to precisely perform tasks.
Instead, we have algorithms that are very good at mimicking tasks. But they aren't very good at composing these tasks to solve more complex problems. We have AIs that are good mimics but are unable to conjure up their own abstractions.
We don't have AIs that can explain how they arrive at their solutions to problems. That's because agents that only learn to mimick do not need to learn to reflect on their own solutions.
But why are machine learning models unaware of their own solutions? Some kinds actually are. These are the kinds that engage in self-play (i.e. GANs, AlphaZero)
• • •
Missing some Tweet in this thread? You can try to
force a refresh
