experiment tracking
In plain wordsKeeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed.
The systematic recording of each training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months. or evaluation run in machine-learning development, together with everything needed to compare and reproduce it: hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., code version, data and environment details, metrics over time such as lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. curves and validation scores, hardware usage, and output artifacts such as checkpointscheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., plots and sample predictions. Tracking tools capture this through a few lines of instrumentation, store it centrally and provide dashboards to filter, chart and compare runs, often integrated with hyperparameter sweepshyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. and a model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use.. Because results depend on many interacting settings, tracking replaces spreadsheets and file names such as 'final_v3_really_final' as the team's memory, and supports audit by showing how a deployed model was produced. Widely used tools include MLflowMLflow, prop. n.: A free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used., Weights & BiasesWeights & Biases, prop. n.: A company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards., TensorBoard, Comet and Neptune.
Check the experiment tracking dashboard before rerunning; someone already tried that learning rate last week.
Origin
Researchers long kept paper or spreadsheet lab notebooks. Dedicated software arrived in the mid-2010s: TensorBoard (2015) visualized training curves, the open-source Sacred tool logged experiment configurations, and trackers such as Comet, Weights & BiasesWeights & Biases, prop. n.: A company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards. and MLflowMLflow, prop. n.: A free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used. Tracking (the last two launched in 2018) made the practice standard.
Related
- MLflowMLflow, prop. n.: A free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used.
- Weights & BiasesWeights & Biases, prop. n.: A company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards.
- model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
- training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
Mentioned in
Jupyter notebookJupyter notebook, n.: An interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work., machine learning pipelinemachine learning pipeline, n.: An automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way., MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself., model registrymodel registry, n.: A central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use., data versioningdata versioning, n.: Keeping a record of every version of the data used to train and test AI models, so that any past result can be traced back to exactly the data that produced it., MLflowMLflow, prop. n.: A free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used., Weights & BiasesWeights & Biases, prop. n.: A company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards.