Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Or Shafran Profile picture
Or Shafran
@OrShafran
Feb 5 • 8 tweets • 3 min read
It's time to look past dictionary learning for decomposing LM activations.

What happens when we instead leverage local geometry?

We find a natural region-based decomposition that yields better steering and localization 🧵 1/ Our shift: From directions (Global) ➡️ regions (Local)

Using Mixtures of Factor Analyzers (MFA) by @ZoubinGhahrama1 we model activation space as a collection of low-rank Gaussians:

📍 Centroid (μ) = region (where you are)

📐 Low-rank subspace (W) = local variation 2/ Image