Thread by @AnthropicAI

Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. In our latest work, we build toy models where the origins of polysemanticity can be fully understood.

If a neural network has 𝑛 feature dimensions, you might intuitively expect that it will store 𝑛 different features. But it turns out that neural networks can store more than 𝑛 features in "superposition" if the features are sparse!
transformer-circuits.pub/2022/toy_model…

For example, if a word embedding has 512 dimensions, one might think it has 512 features (eg. verb-noun, male-female, singular-plural, …). Superposition suggests there may be many more – they're just not exactly orthogonal.

(This is very similar to ideas in compressed sensing!)

Superposition doesn't occur in linear models. It only occurs if one introduces a non-linearity that can filter out noise from superposition "interference". It also requires sparsity: as features become sparser, there is more superposition.

It turns out that whether a feature is stored in a dedicated dimension, not learned at all, or represented in superposition is governed by a discontinuous phase change between three regimes!

Amazingly, features in superposition are organized into subspaces with geometry corresponding to regular polytopes (pentagons, tetrahedra) and other solutions to the Thomson problem. This creates discretized “energy levels” in the amount of dimensionality allocated to features.

We also find that neural nets can do simple computation on features while they're in superposition. This suggests a wild possibility: neural networks may, in some sense, be simulating larger sparse models.

If superposition is widespread, understanding it will be essential for interpretability (especially models like transformers where polysemanticity is dominant). This is a small first step to understanding it.

The ideas in this paper build on lots of previous work, including papers hypothesizing the existence of superposition. It's also connected to ideas in compressed sensing, distributed representations, disentanglement, and other work.

There's a lot more in the paper if you're interested, including phase changes, beautiful geometric structure, surprising learning dynamics, a connection to adversarial examples, and much more!
transformer-circuits.pub/2022/toy_model…

Superposition is easy to explore -- you can play with it on a single GPU! We have a notebook reproducing our basic results to start you off: colab.research.google.com/github/anthrop…