Thread by @IntuitMachine

Von Neumann once told a student who was troubled by the counter-intuitiveness of quantum mechanics:

There are many deep learning practitioners who are also lost in mathematics. The difference of course is that DL folks don't actually have to perform the computations, the network does that for them.

In fact, with methods like architecture search, they can just use the machine to discover the optimal neural architecture.

But neural architecture search is only as good as the hard-coded vocabulary that is combined in a search. Furthermore, it doesn't design higher-level modules like say a Transformer or Capsule block. We can of course add these as a library and thus extend the vocabulary.

In quantum mechanics, there are many alternative interpretations. Interpretations are useful for scientists so that they can devise new hypotheses that leads to new experiments.

The hypothesis of science do not arise out of thin air, they are always based on some intuitive understanding of the domain. An intuitive understanding implies an interpretation of the underlying formal models.

One difference between physics and math is that physics has an interpretation of the mathematics. A physicist looks at an equation and derives a handwavy interpretation of what it means. One very visual kind is the formulation of the least action principle.

In deep learning, there are many practitioners who seek a Bayesian interpretation of the formulations. There are countless papers that talk about priors and posteriors to reason as to why they select elements of their architectures.

I've always argued that the Bayesian perspective is not the only way to interpret the design of neural networks. There are alternative ways of interpretation that can also lead to novel architectures.

In fact, many times researchers employ neuroscience, biological, game theory, physics or ecological metaphors in the design of their algorithms. I think it's impoverished that too many papers are written without explanations using alternative metaphors.

In my case, my favored metaphors come from the semiotic space, and hence it is language-inspired.

But the point I would like to drive home is that you cannot just rely on mathematics to make progress in neural network design. You have to be inspired by richer metaphors.

@threadreaderapp unroll