Thread by @ostojic_srdjan

During my physics undergrad, I have never heard of Singular Value Decomposition (SVD).

Why?

Almost all matrices in physics are symmetric, and in that case SVD reduces to eigenvalue decomposition. 1/n

But for non-symmetric, or even non-square matrices, SVD is really the fundamental tool, and data-scientists have known this well.

https://twitter.com/WomenInStat/status/1285611042446413824

(Original thread by @daniela_witten)

The physicists' bias towards eigenvalue decomposition vs SVD has spilled over into theoretical neuroscience. The Dayan and Abbott book for instance never mentions SVD!

So over the recent years, part of the theoretical neuroscience community (and that part includes me) has been rediscovering how useful SVD is, eg for understanding network models. [The list below is far from exhaustive, please add more examples!]

For instance, basic results on perceptrons can be understood in a simple way using SVD.
tinyurl.com/perceptron-sgd

Dynamics of learning in deep networks can be understood based on SVD.
arxiv.org/abs/1312.6120
pnas.org/content/116/23…
arxiv.org/abs/1809.10374

Non-linear dynamics in recurrent neural networks can be analyzed by starting from the SVD of the connectivity matrix, and keeping dominant terms.
arxiv.org/abs/1711.09672
biorxiv.org/content/10.110…
arxiv.org/abs/2007.02062
arxiv.org/abs/1909.04358
biorxiv.org/content/10.110…

Non-normal transient dynamics in recurrent networks
sciencedirect.com/science/articl…
arxiv.org/abs/1811.07592

Of course neuroscientists with a stat/data-science background have been using SVD for ever, in particular for dimensionality reduction, but there has been a bit of a gap with physicists.

https://twitter.com/WomenInStat/status/1285615926042406913

(Original tweet by @daniela_witten)

At this point, it looks like it’s time to reposition SVD at the basis of the comp neuro curriculum. If you need to catch up, here is a great crash course by @eigensteve
youtube.com/playlist?list=…