inference-time intervention

A steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model. method aimed at making a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. more truthful, which nudges the outputs of a few selected attention headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. in fixed directions each time a token is generated. Linear probeslinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. are first trained on each head's activations to separate true from false answers on a labeled set; the K heads with the most accurate probes are selected, and during generation each selected head's output is moved by α standard deviations along a truth direction — the mean difference between true and false activations, which worked better than the probe weights. In the introducing paper a few hundred labeled examples sufficed to roughly double the TruthfulQATruthfulQA, prop. n.: A test of whether AI models repeat popular myths and misconceptions — questions to which many people, and much of the internet, give a confidently wrong answer. truthfulness score of Alpaca, an instruction-tuned LLaMA, from 32.5% to 65.1%; pushing harder made answers less helpful.

We applied ITI to the top 48 heads and saw fewer misconceptions but more 'I have no comment' answers.