Interesting thread. It's good to see these ideas expressed in such a clear and provocative way, because a lot of people believe them, and they are wrong in an interesting way. THREAD 👇🏻 (post 1 of "until my beer gets cold")
First, what is by now a chestnut "deep learning is great engineering but no science." This is (a) wrong, and (b) contains a fundamental misunderstanding, common in computer science, about what engineering is.
(a) Sure, no theoretical consensus on why deep learning methods work. But the important part was: Before they started working, it seemed clear that they shouldn't. Far too many parameters, not enough control of overfitting. Poor Mr L2 regularization can do only so much.
We are still grappling with how to revise our conceptual understanding of ML. 👉 When important experimental results are not explained by theory, that is a major conceptual advance. A mystery rather than a solution, but finding new mysteries is important.
(b) "It's just engineering". Too many computer science use this phrase to refer to software development tasks that, even when difficult, apply only standard ideas. Thus if you a researcher, engineering is uninteresting, something to minimize
The CS "just engineering" viewpoint is a disservice to the term "engineering". In fact, engineering is a subject of research. Turns out, there are entire university buildings full of people called "Professors of Engineering", and they claim to do research. Who knew?
Engineering is the application of science and math to problems in technology. Some engineering is simple applications following existing paths, that's not research. Other engineering is conceptually new --> that is research.
Some of computer science research is pure mathematics. That's cool. The majority of CS research, though, is engineering: developing new ways to apply theoretical and scientific methods to solve technology challenges.
I forgot about (a) "no new ideas". Lots of the "deep learning tricks" are pretty interesting conceptually: e.g., attention, fancier versions of dropout, learning rate adaptation, deep generative models.
I'd better go, because my wife is about to kill me. Tell you what, if this thread gets more than 100 likes, I'll respond to points about research at Google.
