Thread by @ylecun

Some folks still seem confused about what deep learning is. Here is a definition:

DL is constructing networks of parameterized functional modules & training them from examples using gradient-based optimization.... facebook.com/722677142/post…

This definition is orthogonal to the learning paradigm: reinforcement, supervised, or self-supervised.

Don't say "DL can't do X" when what you really mean is "supervised learning needs too much data to do X"....

....Extensions (dynamic networks, differentiable programming, graph NN, etc) allow the network architecture to change dynamically in a data-dependent way.

[2 year old post about differentiable programming facebook.com/story.php?stor… ]

Don't say "DL is sensitive to adversarial examples" when what you really mean is "supervised ConvNets are sensitive to adversarial examples."

Don't say "DL is biased" when what you really mean "plain supervised learning reproduces the biases in the training data"

Don't say "DL doesn't handle compositionality" when you really mean "this particular architecture doesn't generalize to a number of previously-unseen combinations of parts"

Don't say "DL doesn't do causal inference" when you really mean "a plain, supervised neural net does not spontaneously discover causal relationships."

Don't say "DL doesn't do logical inference" when you really mean "a plain feed-forward neural net can't do long chains of reasoning"

In all of this, the questions are:
(1) what architecture? (it needs to be compatible with backprop)
(2) what learning paradigm and objective function? (RL, SL, SSL...)
(3) what training data and training protocol?

But trainable networks of differentiable modules are here to stay