reversal curse
In plain wordsA quirk of language models: having learned that 'A is B', they often cannot answer the reverse question — they may know who a celebrity's mother is yet fail to say who her son is.
A blind spot of autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. language models: a fact learned in one direction, 'A is B', is often unavailable in the other, 'B is A', unless the reversed statement also appeared in training. Fine-tuned on invented statements, models answered questions phrased in the trained direction yet did no better than chance when asked the reverse, and frontier models showed the same asymmetry on real celebrity–parent pairs. The effect concerns parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.: when both facts are supplied in the promptprompt, n.: The message or instructions you give an AI system — a question, a request, some examples or a document — that it responds to. Clearer prompts generally get better results., models reverse them easily. Explanations point to next-tokennext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers. training, which strengthens associations only in the direction they are read; augmenting data with reversed or permuted text and using bidirectional objectives reduce it.
Classic reversal curse: it knew the drug's side effect but couldn't name the drug from the side effect.
Origin
Named by Lukas Berglund, Owain Evans and colleagues in 'The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'' (September 2023; ICLR 2024). On questions about real celebrities' parents, GPT-4 correctly answered forward questions (such as who Tom Cruise's mother is) 79% of the time, but the reverse questions (whose mother she is) only 33% of the time.
Related
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
- generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
- next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
- in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
- factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up.
Sources
- Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Cooper Stickland, A., Korbak, T. & Evans, O. (2024). The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'. ICLR 2024. arxiv.org/abs/2309.12288
Mentioned in
Owain EvansOwain Evans, prop. n.: A British AI-safety researcher who created the TruthfulQA test and discovered surprising ways language models generalize, such as narrow training making a model broadly malicious., parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.