Interpretability

Overview

Interpretability asks what a model is doing inside, as opposed to what it outputs. For a working engineer the question is rarely philosophical. You need to know whether a classifier is reading the label from a watermark, whether a language model’s refusal is a shallow pattern or a general behaviour, which feature a credit model leaned on for a decision you have to explain, or why a fine-tune broke something it was not trained on. Each of those is an interpretability question with a method attached.

This hub is for engineers who want to use those methods rather than invent them. It covers probing, attribution, mechanistic interpretability and sparse autoencoders, and it spends a section on limits, because the field’s results are easy to over-read. The models being inspected are described in /topics/transformers-and-llms/ and /topics/deep-learning/.

Key ideas

Probing: is the information there?

A probe is a small classifier trained on a model’s internal activations to predict some property: part of speech, the board state in a game, whether the current statement is true. If a linear probe reads the property out with high accuracy, the representation encodes it linearly. Alain and Bengio (2016) used this to show how class information becomes more linearly accessible layer by layer. Two cautions apply. A probe that is too powerful can learn the property itself from noise, so compare against a control task with shuffled labels. And a readable representation is not a used one: the model may encode a property it never acts on.

Attribution: which input mattered?

Attribution methods assign each input feature a share of responsibility for an output. Gradient-based methods (saliency, integrated gradients by Sundararajan et al., 2017) use the derivative of the output with respect to the input. Perturbation methods (SHAP by Lundberg and Lee, 2017, LIME) measure how the output changes when features are removed or replaced. SHAP has a sound footing in cooperative game theory and a fast exact algorithm for tree ensembles, which is why it dominates tabular work. For transformers, attention weights are tempting as a free attribution and are unreliable as one: Jain and Wallace (2019) showed that very different attention patterns can give the same output.

Mechanistic interpretability and sparse autoencoders

Mechanistic interpretability tries to reverse-engineer the computation into named parts: this attention head copies the previous token, that circuit of heads completes “Mr and Mrs Dursley” with the right name. The tool is causal intervention, especially activation patching, where an activation from one run is pasted into another and the output change measures that component’s role. Wang et al. (2022) traced a full circuit for indirect-object identification in GPT-2 small (124 M parameters, 1 k context, MIT licence, 2019).

The obstacle is superposition: a model with fewer neurons than concepts stores concepts as overlapping directions, so a single neuron means several things. A sparse autoencoder (SAE) attacks this directly. It is trained on the residual stream at one layer to reconstruct the activation through a much wider hidden layer with an L1 penalty that keeps only a few hidden units active per token. Those hidden units, the features, tend to be far more interpretable than neurons (Cunningham et al., 2023, and Anthropic’s work on Claude 3 Sonnet in 2024). Gemma Scope (Lieberum et al., 2024) released SAEs for every layer of Gemma 2 2B and 9B, so the method can be tried on an open model.

Limits

Every method above has a failure mode that looks like success. Probes find correlations the model may not use. Attribution scores from different methods disagree on the same input, and some fail simple sanity checks such as being unchanged when the model’s weights are randomised. Circuits found in a 124 M parameter model do not automatically scale. SAE features depend on the width, the sparsity penalty and the random seed, two SAEs trained on the same layer find overlapping but different dictionaries, and a feature’s name is a human’s summary of a few hundred examples. Interpretability results are hypotheses about the model, made more credible by causal interventions and less credible by pretty visualisations. Where a regulation or a customer asks for an explanation of a decision, the honest answer at the time of writing is usually a documented input-attribution method with its known caveats, not a claim to have understood the model.

Where to start

Read /learn/beginner/03-linear-models-from-scratch/ first, because a linear probe is a logistic regression on activations and a linear model’s coefficients are the simplest attribution there is. Then /learn/beginner/07-your-first-neural-network/ for the activations you will be probing and /learn/beginner/08-embeddings-and-similarity/ for the geometry of a representation space. Run SHAP on the tree model from /learn/beginner/05-trees-and-gradient-boosting/, then load a Gemma Scope SAE with SAELens and look at the top-activating tokens for a few features. The /topics/safety-security-and-privacy/ hub covers the audit questions interpretability is asked to answer.

Tools and platforms

Common mistakes

• Showing an attention map as an explanation. It is a diagnostic at best.
• Training a probe with no control task, then concluding the model “knows” the property.
• Running one attribution method and reporting its output as the reason for a decision without a sanity check.
• Naming an SAE feature from its top ten activations and treating the name as ground truth.
• Reading a circuit found in GPT-2 small as a fact about a 70 B model.
• Confusing interpretability with control. Finding a feature for a behaviour does not mean steering on it changes only that behaviour.

Going deeper

• Sundararajan, Taly and Yan, Axiomatic Attribution for Deep Networks, 2017.
• Lundberg and Lee, A Unified Approach to Interpreting Model Predictions, 2017.
• Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, 2022.
• Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models, 2023.
• Lieberum et al., Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2, 2024.
• Elhage et al., Toy Models of Superposition, 2022, the paper that framed the problem SAEs address.