Recent studies have suggested that the earliest iterations of DNN training are especially critical. In our #ICLR2020 paper with @jefrankle and @davidjschwab, we use the lottery ticket framework to rigorously examine this crucial phase of training.
arxiv.org/abs/2002.10365
arxiv.org/abs/2002.10365
@jefrankle @davidjschwab Existing methods can't find winning lottery tickets at init on larger networks. Instead, they only seem to emerge early in training. We exploit this in our experiments by as a causal way to measure the impact of various network properties on this early phase of training.
@jefrankle @davidjschwab First, we found that it is possible to reinitialize lottery tickets of small networks if weights keep the same signs (Zhou et al., 2019), these results do not appear hold on more complex models when we use weights from early in training.
@jefrankle @davidjschwab Second, by using constrained permutations, we observed that the distribution of winning ticket weights after the earliest iterations is already highly non-i.i.d., suggesting that approaches which aim to approximate this distribution from early in training are unlikely to succeed.
@jefrankle @davidjschwab Interestingly, both of these perturbations were well-approximated by simply adding gaussian noise with similar variance to the weights.
@jefrankle @davidjschwab Finally, we measured the data-dependence of this early phase of training, finding that we could well-approximate this phase with self-supervised pre-training, though this required far more training than in the supervised case to achieve equivalent performance (32x longer!).
@jefrankle @davidjschwab We ran these experiments on 5 different models, finding consistent effects: ResNet-20 and ResNet-56 (both designed for CIFAR-10), and ResNet-18, WRN-16-8, and VGG13. Our paper is available on arxiv and we look forward to sharing it with you at #ICLR2020!
arxiv.org/abs/2002.10365
arxiv.org/abs/2002.10365
