Othello-GPT
In plain wordsA small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models.
A GPT-style transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. trained to predict legal next moves in the board game Othello from sequences of move tokens alone, with no access to the board or the rules, used as a test case for whether sequence models learn world modelsworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.. Kenneth Li and colleagues found that the board state could be recovered from its activations with nonlinear probesprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere. but not linear ones, and that editing the probed representation changed the model's predictions in line with the edited board, showing that the representation is causally used. Neel Nanda then showed that the board is linearly represented once squares are described relative to the player to move — 'mine' versus 'theirs' rather than black versus white — supporting the linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.. Othello-GPT became a standard toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work. for sparse-autoencoder and circuit research and a touchstone in debates over whether language models learn only surface statistics.
As in Othello-GPT, a probe recovered the full board state from the residual stream.
Origin
Introduced by Kenneth Li, Aspen Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister and Martin Wattenberg in 'Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task' (October 2022; ICLR 2023). Nanda's linear follow-up appeared as blog posts in March 2023 and in a paper with Andrew Lee and Wattenberg (BlackboxNLP 2023).
Related
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
Sources
- Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H. & Wattenberg, M. (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023. arXiv:2210.13382. arxiv.org/abs/2210.13382
- Nanda, N., Lee, A. & Wattenberg, M. (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. BlackboxNLP 2023.
Mentioned in
world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.