MLCommons
In plain wordsAn industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety.
A nonprofit consortium of chip makers, cloud providers, AI developers and universities that maintains shared benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. for measuring machine learning systems, together with open datasets and data standards. Its MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models. suites are the industry's standard comparisons of hardware and software speed for training (since 2018) and inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. (since 2019): training results report the time to train reference models to a target quality, and inference results report throughput and latency under defined scenarios. Results are submitted by vendors and cloud providers such as Nvidia, AMD, Google and Intel and checked by peer review among submitters. MLCommons also publishes AILuminate (2024), a benchmark grading chatbot responses across categories of hazard, the Croissant metadata format for machine learning datasets, and open datasets such as the People's Speech corpus.
The vendor's press release quoted its MLPerf Inference numbers, but only for the offline scenario.
Origin
Grew out of MLPerf, a benchmarking effort launched in 2018 by researchers from Google, Baidu, Harvard, Stanford, the University of California, Berkeley, and other institutions; MLCommons was established as a nonprofit in December 2020 to host MLPerf and related projects.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
- inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it.
- NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
Mentioned in
AlgoPerfAlgoPerf, prop. n.: A public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one.