Gemma Scope
In plain wordsA free collection of ready-made concept-finding tools released by Google DeepMind for its open Gemma language models, so outside researchers can study their insides without training the tools themselves.
An open suite of sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. released by Google DeepMindGoogle DeepMind, prop. n.: Google's main artificial intelligence laboratory, formed in 2023 by joining DeepMind and Google Brain. It builds the Gemini models and scientific AI systems such as AlphaFold. for its open-weight Gemma language models, intended to make dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. research possible without the large compute cost of training dictionaries. The first release (announced 31 July 2024, technical report August 2024) provided JumpReLU SAEs for every layer and sublayer of Gemma 2 2B and 9B, for selected layers of Gemma 2 27B, and for an instruction-tuned 9B model, at several dictionary widths. Gemma Scope 2 (19 December 2025) covered every layer of all Gemma 3 sizes from 270 million to 27 billion parameters, pretrained and instruction-tuned, adding transcoderstranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes., cross-layer transcoders for the two smallest models and multi-layer crosscoderscrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears.. The releases are browsable on NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on. and have been widely used for research on features, circuitscircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. and steering, including studies of the limitations of SAEs.
Instead of training our own dictionary, we used the Gemma Scope SAE for layer 20 of Gemma 2 9B.
Origin
Described by Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy and colleagues, with Neel Nanda as senior author, in 'Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2' (August 2024). The name joins the Gemma model family to a microscope metaphor.
Related
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- Google DeepMindGoogle DeepMind, prop. n.: Google's main artificial intelligence laboratory, formed in 2023 by joining DeepMind and Google Brain. It builds the Gemini models and scientific AI systems such as AlphaFold.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
Sources
- Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R. & Nanda, N. (2024). Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. arXiv:2408.05147. arxiv.org/abs/2408.05147
- Google DeepMind Language Model Interpretability Team (2025). Gemma Scope 2: helping the AI safety community deepen understanding of complex language model behavior. Google DeepMind blog, 19 December 2025. deepmind.google/blog/gemma-scope-2-helping-the-ai-safety-community-deepen-understanding-of-complex-language-model-behavior
- Google (2025). Gemma Scope. Gemma documentation. ai.google.dev/gemma/docs/gemmascope
Mentioned in
GemmaGemma, prop. n.: Google's family of free-to-download AI language models: smaller relatives of its Gemini systems, compact enough to run on a single computer and, in some versions, on a phone., Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models., NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.