TransformerLens
In plain wordsA free software library that lets researchers open up language models and record, inspect or alter what happens inside them at every step while the model runs.
An open-source Python library, built on PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., for mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. of GPT-style language models. It reimplements many open-weight transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. architectures in a consistent form whose internals are exposed through named hook points, so that users can cache any activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. and attach functions that read, edit, ablate or replace activations during the forward pass — the basic operations of activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network. and direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions.. It loads pretrained weights from public model hubs, standardizes details such as folding layer-normalization weights into adjacent layers, and includes utilities for visualizing attention and analyzing QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.. Widely used in research and teaching since 2022, it inspired companion libraries such as SAELens for sparse autoencoders; alternatives for larger models include nnsight.
Load GPT-2 small in TransformerLens, run with the cache, and plot the layer-9 attention patterns.
Origin
Created in 2022 by Neel Nanda after he left Anthropic's interpretability team, originally under the name EasyTransformer; it was published on PyPI as TransformerLens from December 2022. Later maintained by a community team under the TransformerLensOrg organization on GitHub (as of October 2026, Bryce Meyer and Jonah Larson).
Related
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
- NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
Sources
- Nanda, N. et al. (2022). TransformerLens: A library for mechanistic interpretability of generative language models. GitHub repository. github.com/TransformerLensOrg/TransformerLens
Mentioned in
Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models.