activation addition
In plain wordsA simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
A steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model. method that builds a steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it. from a single pair of contrasting prompts and adds it to the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. during generation, with no optimization or training. Both prompts (for example 'Love' and 'Hate') are run through the model; the difference of their activations at a chosen layer is multiplied by a coefficient and added at the first token positions of a new prompt's forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.. On GPT-2-XL and later models such vectors redirected the sentiment, topic or style of completions, while fluency, and the handling of text unrelated to the steered property, mostly survived. ActAdd showed that behaviorally meaningful directions can be found with almost no data or compute, supporting the linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.; contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery. later averaged over many pairs for more reliable vectors.
A single 'Love' minus 'Hate' ActAdd vector was enough to turn GPT-2-XL's hostile completions affectionate.
Origin
Introduced by Alexander Matt Turner and colleagues, first in a May 2023 LessWrong post on steering GPT-2-XL and then in an August 2023 arXiv paper (later retitled 'Steering Language Models With Activation Engineering'). It grew out of the same group's early-2023 'cheese vector' experiments on a maze-solving reinforcement-learning agent.
Usage note
'Activation addition' is sometimes used generically for any additive steering; ActAdd names the specific single-pair method.
Related
- steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
- activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
- contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery.
- activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
Sources
- Turner, A. M. et al. (2023). Activation Addition: Steering Language Models Without Optimization. arXiv:2308.10248. arxiv.org/abs/2308.10248v4
Mentioned in
activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery., directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does., steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.