Thread by @JonAMichaels

#tweeprint time!
How do we generate the right muscle commands to grasp objects? We present a neural network model that replicates the vision to action pipeline for grasping objects and shows internal activity very similar to the monkey brain.

biorxiv.org/content/10.110…

Monkeys grasped and lifted many objects while we recorded neural activity in the grasping circuit (AIP, F5, & M1 - see original paper elifesciences.org/articles/15278). All of these areas have been shown to be necessary for properly pre-shaping the hand during grasping.

We show that the advanced layers of a convolutional neural network trained to identify objects (Alexnet) has features very similar to those in AIP, and may therefore be reasonable inputs to the grasping circuit, while muscle velocity was most congruent with activity in M1.

Based on these results, we constructed a modular neural network model (top of thread) to transform visual images of objects into the muscle kinematics necessary to grasp them. Activity in the model was very similar to neural activity while monkeys completed the same task.

We tested how different neural network architectures and regularizations affected these results, finding that modular networks with visual input best matched neural data and showed similar inter-area relationships as in the brain.

Importantly, networks used simple computational strategies for maintaining, reorganizing, and executing movements, relying on a single fixed point during memory and a single fixed point during movement.

This simple strategy allowed networks to generalize well to novel objects, even predicting the real neural activity for these objects, providing a powerful predictive model of how flexible grasp control may be implemented in the primate brain!

Thanks to my fantastic collaborators @SSchaffelhofer, Andres Agudelo-Toro, Hans Scherberger.

Thanks to the @DPZ_eu for supporting the work, and @shenoystanford and @andpru for letting me keep working on the paper when I had more important things to do.

Thanks to everyone who gave great feedback along the way, including @dann_benjamin, @bsauerbrei1, @SaurabhsNeurons, @JAlGallego , @mattperich, @MattGolub_Neuro, and many others.