Skip to content
Last Dictionary
Plane
Interpretability & SteeringTraining & Post-trainingSafety & Alignment

model editing

nounAlso: knowledge editing, fact editing, weight editing, edit a model

In plain wordsChanging one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.

  1. Methods that alter a trained model's behavior on a targeted set of inputs — most often a factual association of the form (subject, relation, object) — while leaving its other behavior unchanged and without full retraining. Locate-then-edit methods identify the responsible parameters and modify them in closed form (ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights., MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers.); meta-learned editors predict weight updates from fine-tuning gradients (KnowledgeEditor, MEND); memory-based methods store edits outside the weights and route matching inputs to them (SERAC, GRACE). Edits are judged on reliability (the edit takes), generalization to paraphrases, locality (unrelated facts are unchanged) and portability (consequences of the new fact also update). Known weaknesses include poor propagation to logically entailed facts, side effects that accumulate over many sequential edits, and potential misuse to plant false information. Model editing is related to machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again. and contrasts with steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., which changes behavior at inference time rather than in the weights.

    After model editing, the assistant named the new prime minister but still gave the old one's party.

Origin

Early work includes Anton Sinitsin and colleagues' 'Editable Neural Networks' (ICLR 2020) and David Bau and colleagues' rewriting of generative image models (ECCV 2020). For language models, Nicola De Cao, Wilker Aziz and Ivan Titov's KnowledgeEditor (2021), Eric Mitchell and colleagues' MEND (2021–2022) and Kevin Meng and colleagues' ROME (2022) established the field.

Related

See also
  • ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.
  • MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers.
  • knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.
  • causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
  • machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again.
  • steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it.
  • parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
Contrast with
  • fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.

Sources

  1. Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. D. (2022). Fast Model Editing at Scale. ICLR 2022.
  2. Meng, K., Bau, D., Andonian, A. & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262. arxiv.org/abs/2202.05262

Mentioned in

steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact., MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers., parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book., ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.