training–serving skew
In plain wordsA mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.
A gap between what a model sees, or how well it does, when it is trained and when it is used in production, caused by the system itself rather than by changes in the world. Typical sources are separate code paths that compute the same feature differently (for example, a batch job for training and a real-time service in another language for serving); features that use information not available at prediction time; stale or delayed feature values; mismatched preprocessing such as tokenization or image resizing; and feedback loops in which the model's own decisions alter later training data. Skew is insidious because offline evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews. looks good while live quality silently degrades. Prevention relies on sharing one feature-transformation code base, logging the exact features used at serving time and training on those logs, feature storesfeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live. with point-in-time-correct joins, and validators that compare training and serving data distributions.
The live model underperformed because of training–serving skew: the mobile client rounded ages before sending them.
Origin
No first use has been traced; the earliest widely cited uses come from Google engineering writing of the mid-2010s. Martin Zinkevich's widely circulated 'Rules of Machine Learning' (c. 2016) devotes several rules to it, and the TFX platform paper (Denis Baylor and colleagues, KDD 2017) describes automated skew detection. The term spread with MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself. tooling and feature-store marketing.
Usage note
Distinct from data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., which arises because the world changes after deployment; skew is built into the system and can be present from the first day.
Related
- feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.
- data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
- data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
- machine learning pipelinemachine learning pipeline, n.: An automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
Mentioned in
data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints., machine learning pipelinemachine learning pipeline, n.: An automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way., feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live., machine learning technical debtmachine learning technical debt, n.: The hidden future cost of quick shortcuts in machine-learning systems — tangled data dependencies, messy glue code, unchecked feedback loops — that later make them hard and risky to change., shadow deploymentshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over.