model deployment
In plain wordsPutting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong.
The process of releasing a trained model into an environment where it serves real users or systems, and the resulting production configuration. Deployment involves packaging the model with its preprocessing code and dependencies (often as a container image or in a portable format such as ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.); choosing a serving pattern — online model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost. behind an API, batch scoringbatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., streaming, or execution on users' devices; provisioning hardware; and wiring up logging and monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.. Because offline test scores do not guarantee live behavior, releases are usually staged: shadowshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over. runs, canarycanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone. releases to a small share of traffic, A/B testsA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better., and blue-green switches between two identical environments limit the damage a bad model can do and allow fast rollback to the previous version, typically fetched from a model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use..
We deployed the new ranking model to five percent of traffic before rolling it out globally.
Origin
The usage comes from software release practice. Early portable formats for statistical models included PMML (Predictive Model Markup Language, first drafted in 1997); dedicated serving systems such as TensorFlow Serving (2016) and container-based platforms later turned deployment into a routine, automated step of MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself..
Usage note
Often conflated with model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., the runtime that answers requests; deployment also covers packaging, release strategy and rollback. Under the EU AI ActEU AI Act, prop. n.: The European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models., a 'deployer' is any person or body that uses an AI system under its authority (other than for purely personal use), not the engineer who releases it.
Related
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- shadow deploymentshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over.
- canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone.
- A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.
- MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
- model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
Mentioned in
A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better., machine learning engineermachine learning engineer, n.: An engineer who turns machine-learning ideas into working systems: preparing data, training and testing models, and running them reliably inside real products., MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself., model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone., ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer., shadow deploymentshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over.