MLPerf
In plain wordsA standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models.
An industry benchmark suite, maintained by the MLCommons consortium, that measures how fast machine-learning hardware, software and systems perform fixed tasks to a fixed quality target. MLPerf Training times how long a system takes to train reference models — for example image classification, recommendation, and large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. pretraining and fine-tuning — to a target accuracy; MLPerf Inference measures throughput and latency of trained models under defined serving scenarios, and further suites cover mobile, edge, storage and power. Submissions follow rules on permitted optimizations, are reviewed by other submitters and are published in periodic rounds. Nvidia submits the most results, with Google, AMD, Intel and others participating; critics note that the benchmarks lag the newest models and reward vendors able to invest heavily in tuning.
Origin
Launched in May 2018 by researchers from Baidu, Google, Harvard, Stanford, the University of California, Berkeley and other institutions and companies; the first training results appeared that December. The nonprofit MLCommons was formed in 2020 to run it.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
- AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models.
- FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
Sources
- Mattson, P. et al. (2020). MLPerf Training Benchmark. Proceedings of Machine Learning and Systems 2 (MLSys 2020).
- Reddi, V. J. et al. (2020). MLPerf Inference Benchmark. Proceedings of ISCA 2020.
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., AlgoPerfAlgoPerf, prop. n.: A public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one., MLCommonsMLCommons, prop. n.: An industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety.