Triton

Programming language

An open-source, Python-embedded language and compiler for writing GPU kernelskernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU. at the level of blocks (tiles) of data rather than individual threads. The programmer writes a kernel as operations on whole tiles (loads, matrix multipliesmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing., reductions) and leaves the low-level decisions, such as when to use tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., how work is scheduled, how threads share on-chip memory and how global-memory accesses are combined, to the compiler, which generates code for Nvidia, AMD and other accelerators through the MLIR and LLVM compiler frameworks. Fused, high-performance kernels for attention, normalization or quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. matrix multiplication can thus be written in tens of lines rather than hundreds of lines of CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI.. Triton is the default code generator of PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.'s TorchInductor compiler (PyTorch 2.0, 2023) and is widely used for research kernels, making it a key route to hardware portability.

We wrote the fused attention kernel in Triton, and it ran within 10% of the hand-tuned CUDA version.

Inference serving

Short for Nvidia's Triton Inference Server, an open-source system for serving models from many frameworks with dynamic batching and multi-GPU support, since rebranded as NVIDIA Dynamo-Triton. It is unrelated to the language despite the shared name.