Discover and read the best of Twitter Threads about #uai2020
Most recents (1)
Really existed about our #UAI2020 paper with @IAugenstein & @vageeshsaxena.
TX-Ray: interprets and quantifies adaptation/transfer during self-supervised pretraining and supervised fine-tuning -- i.e. explores transfer even without probing tasks. #ML #XAI
arxiv.org/abs/1912.00982
TX-Ray: interprets and quantifies adaptation/transfer during self-supervised pretraining and supervised fine-tuning -- i.e. explores transfer even without probing tasks. #ML #XAI
arxiv.org/abs/1912.00982
TX-Ray adapts the activation maximization idea of "visualizing a neuron's preferred inputs" to discrete inputs - NLP. With a neuron as an 'input preference distribution' we can measure neuron input-preference adaptation or transfer. This works for self- & supervised models alike.
We analyzed how neuron preferences are build and adapted during: (a) pretraining, (b) 'zero-shot' application to a new domain, and (c) by supervised fine-tuning.
(a) Confirms that: pretraining learns POS first, as @nsaphra showed, and that preferences converge like perplexity.
(a) Confirms that: pretraining learns POS first, as @nsaphra showed, and that preferences converge like perplexity.
