representation fine-tuning
In plain wordsA way of adapting a language model to a new task by training small, targeted edits to its internal signals instead of changing its weights, often with far fewer adjustable numbers than other methods.
A family of parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money. methods that leave every weight of a model untouched and adapt it instead by training small edits applied to its hidden states at selected layers and token positions. The main variant, LoReFT (low-rank linear subspace ReFT), edits each targeted hidden state only within a low-dimensional subspace spanned by the orthonormal rows of a learned matrix R, replacing the state's coordinates in that subspace with a learned linear function of the state. Because interventions touch only a few positions, LoReFT trained 15 to 65 times fewer parameters than LoRAlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves. while matching or beating it on commonsense reasoning, instruction following and language-understanding benchmarks, though not on arithmetic reasoning. ReFT sits between activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., which applies fixed hand-derived edits, and conventional fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., which learns weights from data.
A rank-4 LoReFT intervention on four prompt positions was enough to teach the model our output format.
Origin
Introduced by Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning and Christopher Potts of Stanford University in 'ReFT: Representation Finetuning for Language Models' (arXiv, April 2024; NeurIPS 2024), building on interchange interventions from causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped. research.
Related
- parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money.
- low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.
- activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
- causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
- steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.
Sources
- Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D. & Potts, C. (2024). ReFT: Representation Finetuning for Language Models. NeurIPS 2024. arXiv:2404.03592. arxiv.org/abs/2404.03592
Mentioned in
activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.