Excited to share our paper arxiv.org/abs/2105.12221 on neural net overparameterization to appear at #ICML2021 💃🏻We asked why can’t training find a minimum in mildly overparameterized nets. Below, a 4-4-4 net can achieve a zero-loss, but any of 5-5-5 nets trained with GD can not🤨 We investigated the training failures in mild overparameterization vs. successful training in vast overparameterization from a simple perspective of permutation symmetries!