Just read through the @distillpub interpretability blog post from @ch402 and others. Stunning (and fun!) visualizations, but I wonder: what did these visualizations actually teach us about these networks? What do we know now that we didn't know before? distill.pub/2018/building-…
Saliency, activation maximization, etc. give us the impression of understanding, but it's often extremely difficult to express the conclusion of this "understanding." Absent a falsifiable hypothesis and rigorous quantification, can we actually say we've learned anything?
I wonder if all of these visualization methods are actually Rorschach ink blot tests for ML researchers. We see in them what we want to see, which may often just be the easiest to understand explanation.
Also worth noting that while we only looked at class selective neurons, our recent @ICLR18 paper found that easily interpretable neurons were no more important than confusing neurons, which is at odds with the motivation behind many of these techniques. openreview.net/forum?id=r1iuQ…
And related to the move towards confusing (or "mixed selectivity" neurons) in neuroscience, with lots of recent results showing that they carry tons of information. E.g., nature.com/articles/natur… from @MattiaRigotti and also nature.com/articles/nn.44… among many others.
