Simulating a primary visual cortex at the front of CNNs improves their robustness to image perturbations. #AI still has a lot to learn from #neuroscience. Work co-lead with @joeldapello. Also @martin_schrimpf @JamesJDiCarlo @GeigerFranziska @neurobongo 1/N biorxiv.org/content/10.110…
This work was the result of an unexpected collaboration. Joel discovered that the ability of CNNs to explain V1 responses was correlated with their adversarial robustness. Particularly, adversarially trained models [@aleks_madry] had the most V1-like representations. 
For those who don't know, CNNs are easily fooled by imperceptible perturbations explicitly crafted to induce mistakes (adversarial attacks). Currently, the best defense is to explicitly train models to be robust to these attacks which has a very high computational cost.
Simultaneously, I was building benchmarks to evaluate how well CNNs explain V1 single-neuron properties and found that a Gabor filter model constrained by empirical data [@DarioRingach] still outperformed all CNNs tested. Could we improve current CNNs by engineering a better V1?
We went on to develop VOneNets, hybrid CNNs with a fixed-weight V1 model front-end. Take any standard CNN architecture, remove its first block, and replace it by our empirically-constrained Gabor filter bank model of V1.
VOneNets based on 3 CNN architectures (ResNet50, CORnet-S, AlexNet) maintained similar clean accuracy, and were considerably more robust than the standard models! This increased robustness persisted for perturbation strengths that left the standard models near chance level.
Without any expensive adversarial training - just placing a V1 at the front! Surprisingly, VOneNets were not only better than standard models, but they outperformed SOTA on a conglomerate benchmark of perturbations with adversarial attacks and common image corruptions.
Which components of VOneNets are responsible for this gain in robustness? Interestingly, removing any part of the V1 model resulted in less robust models, suggesting that they all interact synergistically!
Removing V1 stochasticity had the largest single effect on perturbation accuracy. However, adding only V1 stochasticity to ResNet50 resulted in only 1/3 of the improvement in robustness. This suggests that V1 stochasticity and features interact nonlinearly to improve robustness!
Also, the vast majority of the improvement in robustness is not due to the stochasticity during the attack or inference. V1 stochasticity during training makes the downstream layers learn more robust representations.
We have very excited about these results as they clearly show that there is plenty of opportunities for a neuro-inspired AI. We feel that we are only tapping the tip of the iceberg and a lot of future work needs to be done!
This work was partly inspired by the fantastic study showing that regularizing CNNs to develop mouse V1-like representations leads to gains in robustness in gray-CIFAR. Zhe Li, @wielandbr @bethgelab @sinzlab @xaqlab @AToliasLab and others. arxiv.org/abs/1911.05072
