,
16 tweets,
7 min read
Read on Twitter
We study bias-variance dilemma in class, but reading the paper gives important historical perspective. You'll see that many of the 'unreasonable effectiveness' and 'surprising findings' are predicted in this paper. Thread..(1) dam.brown.edu/people/geman/H… 
The paper asks the very important question -- given unlimited time (or "compute", as it is called these days) and training data, what sorts of tasks can be learned. Are there limits to such brute-force approaches? (2)
Clearly the authors didn't have the right scale to measure excitement :), but the promise of neural nets is correctly identified. (3)
The core of the dilemma is explained well in the introduction -- it is about the large training size for model-free methods. The paper also notes that parallel architectures and fast hardware do not help with this aspect of the problem. (4)
I find it interesting that midway through the paper the authors come back to re-emphasize this point. (Were the worried that people would forget in all that excitement?) (5)
"...many recent neural models are important tools for wide-ranging applications". Of course they are, and it is interesting that this was recognized even back then. They didn't predict the fake news application though.... (6)
..probably because they used "email" back then. But look at how terrific this paragraph is! It is interesting to note that many of these are already done, and there is intense discussion (thanks to @GaryMarcus ) on the adequacy of these models for cognition.
Justification for the optimism about neural networks: there won't be any need for preprocessing. You can just train from pixels, and with enough compute and enough data, they will approximate the best possible for the task at hand. (8)
Whether the optimism is realistic hinges on just one thing: can training samples be large enough? They then examine the humble handwritten character recognition problem. (9)
For "complex" perceptual tasks such at this, "sufficiently large training set" exists only in theory!! Even for the supposedly simple handwritten character recognition problem? (10)
That was in 1992. The 2018 paper from @bethgelab and @wielandbr made the following observation. arxiv.org/pdf/1805.09190… (11)
The @LakeBrenden and @gershbrain paper on 'Building machines that learn and think like people' identifies character recognition as a grand challenge for AI! So, the bias-variance dilemma authors are still right. cims.nyu.edu/~brenden/LakeE… (12)
Back to the original paper, the authors have a nice paragraph on interpolation vs extrapolation, and generalizing in a 'non-trivial sense'. (13)
This is another interesting paragraph about the excitement about having solved the credit assignment problem and about the ability to form internal representations. (14)
The authors recognize that for many applications acceptable performance can be achieved without unrealistic numbers of training examples. This aspect is probably more obvious now than it was at that time. And many many applications remain. (15)
The authors conclude by stating that identifying the right preconditions for learning is as important as learning itself. (16)
