What should be baffling for so many but isn't mentioned enough is how a neural network that is based on continuous mathematics leads to things like GPT-3 that works on discrete tokens.
3 years ago, I wrote a rebuttal blog post that argued why Deep Learning could be applied to NLP. This was in response to a post making the rounds that it was impossible. medium.com/intuitionmachi…
The argument against Deep Learning was that it is based on continuous function and thus cannot be applied to non-continuous things like words: linkedin.com/pulse/google-h…
My arguments however back then were based on empirical evidence in the field. DL in NLP was already paying dividends in 2017. What is missing of course is, why does it work?
What is it about language that obviously looks discrete, but surprisingly can be processed by something that works only in the continuous?
The problem is that the mathematical tools we employ are ones that originate from Newton. There is a gap in our folk knowledge as how the discrete and the continuous interact.
A little over 100 years ago, there was a debate as to which was more fundamental, logic or mathematics. In the eyes of the public, it was logic that won that debate.
In 1931, the supremacy of logic over mathematics was falsified. See: en.wikipedia.org/wiki/G%C3%B6de…
Reality is mathematical, in fact, as Turing has revealed is that reality is computational.
It was Bertand Russell who proposed logic over mathematics. On the other side of that debate was the American C.S.Peirce.
It is C.S. Peirce who came up with quantification that we use in predicate calculus. C.S.Peirce in fact had a graphical notation that expressed this: 
The greatest of all American thinkers, C.S.Peirce, is unfortunately not well known. We have here a person who revealed the possibility of NAND and NOR gates as universal building blocks for logic, yet few know of him.
The artificial logic systems (i.e. computers) that we benefit from today, he understood how they would be built 100 years ago. In fact, he was prescient enough to foretell of the demise of GOFAI. This was over a half a century before A.I. was coined.
He formulated that human cognition consisted of induction, deduction and abduction. The first two we have artificial version of today. That is Deep Learning and Symbolic logic.
He was sufficiently way beyond his time that we need to ask. What did he understand about the interplay with the discrete and the continuous? Surely this paradox couldn't avoid his scrutiny.
How does C.S.Peirce thinking lead to an understanding of how continuous functions leads to discrete language understanding?
Let's explore how to frame current #transfomer language models in the framework of Pericean semiotics. gum.co/empathy
@threadreaderapp unroll
