machine learning pipeline
In plain wordsAn automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way.
A sequence of automated, dependent stages that turns data into a deployable model and, in production, turns inputs into predictions. A training pipeline typically ingests and validates data, computes or retrieves features, splits datasets, trains and tunes the model, evaluates it against thresholds and baselines, and registers the resulting artifact; an inference pipeline applies the same preprocessing to live or batch inputs before calling the model. Stages are usually expressed as a directed acyclic graph run by an orchestrator (Airflow, Kubeflow Pipelines, TFX, Metaflow, Dagster and others), with each step's inputs, outputs and code versions logged so that runs are reproducible and can be retriggered on a schedule or when new data arrives. Within a single library the term also denotes an object that chains preprocessing steps and a model so they are fitted together, as in scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way.'s Pipeline, which helps prevent data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints. from test datatest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam. into training.
The pipeline failed at the validation step because an upstream column had silently switched units.
Origin
Chaining processing stages is an old idea (Unix pipes date from 1973). In machine learning, scikit-learn's Pipeline class (early 2010s) and Apache Spark's ML Pipelines API (2014–2015) chained transformers and estimators; Google's TFX platform (described in 2017) and Kubeflow Pipelines (2018) made orchestrated, end-to-end production pipelines a central object of MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself..
Usage note
Distinct from a data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., which moves and transforms data for any purpose; from the 'pipeline' helper in Hugging Face TransformersTransformers, prop. n.: A free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code., a wrapper for running inference; and from pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy., a way of splitting one model across devices.
Related
- MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
- feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.
- experiment trackingexperiment tracking, n.: Keeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed.
- model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use.
- training–serving skewtraining–serving skew, n.: A mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.
Sources
- Baylor, D. et al. (2017). TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. Proceedings of KDD 2017.
Mentioned in
data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., Jupyter notebookJupyter notebook, n.: An interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work., MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself., model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live., machine learning technical debtmachine learning technical debt, n.: The hidden future cost of quick shortcuts in machine-learning systems — tangled data dependencies, messy glue code, unchecked feedback loops — that later make them hard and risky to change., MLflowMLflow, prop. n.: A free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used., training–serving skewtraining–serving skew, n.: A mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.