shadow deployment
In plain wordsQuietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over.
A release technique in which a candidate model receives a copy of live production traffic and makes predictions that are logged but not acted upon, while the incumbent model continues to serve users. Comparing the two on identical real inputs reveals disagreements, errors, latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. and cost under realistic load, crashes on unusual inputs, and training–serving skewtraining–serving skew, n.: A mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised., all without user-facing risk. Shadowing has limits: users never react to the shadow model's outputs, so effects on engagement or downstream behavior require an A/B testA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better. or canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone., and accuracy can be judged only when labels arrive or through human or model review of disagreements. It also multiplies inference cost for the mirrored traffic, and the shadow system must be prevented from causing side effects such as sending messages or writing to production databases.
We ran the new claims model in shadow deployment for a month and reviewed every case where it disagreed with the old one.
Origin
Adapted from web-operations practices such as dark launching, used by Facebook engineers around 2008, and traffic mirroring. 'Shadow mode' became widely known in 2016, when Tesla said it was running Autopilot software silently in customers' cars to compare its decisions with those of drivers.
Usage note
Distinct from a canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone., in which a small share of users do receive the new model's outputs.
Related
- canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone.
- A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.
- model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
- canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone.
Mentioned in
A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better., model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong., staged deploymentstaged deployment, n.: Releasing a new AI system step by step — first to testers, then to limited groups, then widely — so that problems can be spotted and fixed before everyone has it., canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone.