,
7 tweets,
3 min read
Read on Twitter
A few nuggets from @geoffreyhinton's talk from earlier today at the #ml4h unconference. First up, the distinction between statistics and AI (and presumably ML by implication). Overall, I think these are pretty clean contrasts: 
On interpretability: It's going to be very hard to explain the contribution of 1000s of statistical regularities that a net uses to produce a decision. Humans are really bad at explaining their decisions (most are post-hoc), so we shouldn't hold neural nets to a double standard
Related to interpretability: Regulating neural nets should be based on how it performs, not based on how it works. This is what we do with people too.
On the (unexpected) connection between neural nets and decision trees. Decision trees are inefficient because the leaf nodes operate on a very small fraction of the full data. DNNS with ReLU can be thought of as a kind of decision tree with massive amounts of parameter sharing.
On adversarial examples. These are really bad, but highlight that neural nets do not see the world in the same way that humans do. A possible fix - impose a coordinate system so that a neural net can see the same object in different ways under different coordinate frames.
Example: Humans can see the white box in two different ways - a diamond or a rectangle - depending on the coordinate frame you impose on the box. Different coordinate frames make you sensitive to different kinds of changes in the object.
To sum up, Hinton has this amazing way of describing something that makes it not only sound clearly correct, but intuitive and obvious. It was great to see how he thinks about things first hand.
