CUDA
In plain wordsNvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI.
Nvidia's proprietary toolchain and execution model that lets programmers run arbitrary computations, not just graphics, on its GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.. It comprises extensions to C and C++ (with Fortran and Python bindings) in which a programmer writes a kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU. executed by thousands of threads — organized into blocks that share fast on-chip memory and synchronize, and grids of blocks; a compiler (nvcc), runtime and driver; and libraries such as cuBLAS for linear algebra, cuDNN for neural-network primitives and NCCL for multi-GPU collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.. Frameworks such as PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems. and JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips. call these libraries, so most deep-learning code runs on CUDA without users writing any. The maturity of this software and its developer base — often called Nvidia's CUDA moatmoat, n.: A lasting business advantage that keeps competitors out, like a castle's moat; in AI, the question of whether any company can stay ahead when rivals and free models catch up quickly. — is widely seen as a main reason competitors' hardware has struggled to displace Nvidia; ROCmROCm, prop. n.: AMD's free, open-source software for programming its graphics and AI chips — its answer to Nvidia's CUDA, letting AI software run on AMD hardware., TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts. and AI compilersAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible. aim to loosen the lock-in.
The custom CUDA kernel fused the three operations and halved the step time.
Origin
Developed at Nvidia from 2004 by a team including Ian Buck, whose Stanford Brook language was its basis, and John Nickolls. Introduced in November 2006 alongside the G80 GPU (GeForce 8800), with the first public SDK in February 2007; Nvidia later stopped expanding the acronym. Its use for AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward. (2012) and the release of cuDNN (2014) bound deep learning to it.
Usage note
Also used generically for GPU code ('a CUDA kernel'). How durable the CUDA advantage will be is debated, as higher-level compilers increasingly target several hardware back ends.
Related
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- GPGPUGPGPU, abbr.: Using a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical.
- ROCmROCm, prop. n.: AMD's free, open-source software for programming its graphics and AI chips — its answer to Nvidia's CUDA, letting AI software run on AMD hardware.
- TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts.
- AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible.
- tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
- kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU.
- ROCmROCm, prop. n.: AMD's free, open-source software for programming its graphics and AI chips — its answer to Nvidia's CUDA, letting AI software run on AMD hardware.
Sources
- Nickolls, J., Buck, I., Garland, M. & Skadron, K. (2008). Scalable Parallel Programming with CUDA. ACM Queue 6(2).
Mentioned in
GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., Jensen HuangJensen Huang, prop. n.: The Taiwanese-American co-founder and chief executive of Nvidia, whose graphics chips became the main engine for training and running modern AI., NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., AMDAMD, prop. n.: An American chip company that makes computer processors and graphics chips, and Nvidia's main rival in the chips used for artificial intelligence., HuaweiHuawei, prop. n.: A Chinese technology giant, known for telecom equipment and phones, that designs China's leading homegrown AI chips despite US trade restrictions., kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., GPGPUGPGPU, abbr.: Using a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical., NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., ROCmROCm, prop. n.: AMD's free, open-source software for programming its graphics and AI chips — its answer to Nvidia's CUDA, letting AI software run on AMD hardware., TensorRTTensorRT, prop. n.: NVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest., TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models., TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts., XLAXLA, abbr.: A compiler from Google that turns AI programs into fast code for chips such as TPUs and GPUs, mainly by merging many small steps into fewer, bigger ones.