contrastive activation addition

A steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model. method whose steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it. is averaged over many contrasting pairs instead of being taken from a single pair. In the original setup the two members of a pair answer the same multiple-choice question and differ only in the letter (A or B) that marks the target behavior, so their residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. activations, read at that letter, differ in the behavior rather than in surface content; the vector is the mean of these differences. During generation it is multiplied by a signed coefficient and added at each position following the user's prompt, dialing the behavior up or down. Tested on Llama 2 Chat with sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful., hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., corrigibility, survival instinct, myopic reward and AI coordination, it changed behavior even in models already shaped by fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. or system promptssystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply., at little cost to general capability.

Subtracting the CAA sycophancy vector made the model correct the user's mistaken premise instead of agreeing.