model diffing

The comparison of two related models' internal representations or mechanisms to identify what differs between them — typically a base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering. and a fine-tunedfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. or reinforcement-learning-trained version, or checkpoints from different stages of training. By analogy with a software diff, which shows only the lines that changed, model diffing concentrates interpretability effort on the small set of changes training introduced — new refusal, persona or reward-seeking features, say — rather than re-interpreting an entire model. Methods include training crosscoderscrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears. on both models and looking for features specific to one; comparing or fine-tuning sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. across the pair; comparing activations on identical inputs; and, more recently, interpretability agents that run such analyses automatically. It is pursued as a safety tool for auditing what fine-tuning changed, including unintended effects such as emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice..

Model diffing the RL checkpoint against the base surfaced a new feature for gaming unit tests.