Nils Rethmeier Profile picture
May 20, 2020 6 tweets 4 min read Read on X
Really existed about our #UAI2020 paper with @IAugenstein & @vageeshsaxena.

TX-Ray: interprets and quantifies adaptation/transfer during self-supervised pretraining and supervised fine-tuning -- i.e. explores transfer even without probing tasks. #ML #XAI
arxiv.org/abs/1912.00982 Image
TX-Ray adapts the activation maximization idea of "visualizing a neuron's preferred inputs" to discrete inputs - NLP. With a neuron as an 'input preference distribution' we can measure neuron input-preference adaptation or transfer. This works for self- & supervised models alike. Image
We analyzed how neuron preferences are build and adapted during: (a) pretraining, (b) 'zero-shot' application to a new domain, and (c) by supervised fine-tuning.

(a) Confirms that: pretraining learns POS first, as @nsaphra showed, and that preferences converge like perplexity. Image
(b) We then apply the pretrained language model to new domain texts absent retraining (zero-shot), telling us which neurons transfer - i.e. show similar preference on the new and pretrained corpora. Differing preference marks unknown inputs and 'unhelpful' pretrained neurons. Image
(c) When fine-tuning the pretrained LM to a supervised task, we see that: (i) many pretrained neurons are forgotten (deactivated) and that (ii) previously unpreferred, specialized neurons are added.

Pruning deactivated (i)-neurons raises generalization - i.e. XAI guides pruning. Image
In summary, our method (named TX-Ray) allows one to explore (knowledge abstraction) transfer at neuron-level, which works for (self)-supervised models and even if annotated probing tasks are unavailable.

Implementation details and examples can be found at github.com/copenlu/tx-ray

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Nils Rethmeier

Nils Rethmeier Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(