If this result holds up generally, merging networks trained on different parts of data looks feasible. Huge implications - privacy through federated learning, parallelization of learning, merging of em shards!
https://twitter.com/SamuelAinsworth/status/1569719494645526529
The basic idea is simple: you can permute hidden layer neurons, so there are actually far fewer internal models than it looks. Training gets to one, with linear mode connectivity. Can hence interpolate differently trained networks if one is careful. 
If each layer has N neurons and weights can have K values there are N! permutations and K^(N^2) weight matrices. ln K^(N^2)/N! = N^2 ln K - N ln N + N + O(ln N). So wide networks have room for more possible models, but SGD consistently only seems to find one basin of attraction.
To me the surprise was that the merging was relatively cheap; I had expected it to be hard. But greedy approximation algorithms win anyway.
Looking at my own @AIObjectives project, this suggests that markets may get robustness and convergence properties because companies are permutable (company fungibility?) and market process like SGD converges on a global, easy to find optimum. We'll see.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
