tensor core
In plain wordsA small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
A dedicated matrix multiply–accumulate unit inside a GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.'s streaming multiprocessors that computes D = A × B + C on small tiles of low-precision numbers in one operation, instead of one fused multiply–add per thread per cycle. The first generation (Nvidia's Volta V100, 2017) multiplied 4 × 4 FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values. matrices with FP16 or FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over. accumulation, raising the V100's dense FP16 throughput to 125 teraFLOPS. Later generations added INT8 and INT4 (Turing, 2018); bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate., TF32 and 2:4 structured sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation. (Ampere, 2020); FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. (Hopper, 2022); and FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. with hardware microscalingmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate. (Blackwell, 2024). Nearly all of a modern accelerator's advertised AI throughput comes from such units, so kernels must be shaped — tile sizes, data layouts, precision — to keep them busy. AMD's equivalent is the Matrix Core, and Google's TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. uses larger systolicsystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently. matrix units.
Origin
Introduced by Nvidia with the Volta architecture and Tesla V100, announced on 10 May 2017; the name echoes the tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way. operations of deep-learning frameworks. Google's TPU, in service from 2015, had already shown the value of a dedicated matrix unit.
Usage note
Capitalized 'Tensor Core' is Nvidia's product term; generic use covers similar matrix units in other chips. Not to be confused with a TPU core or a CPU core.
Related
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- systolic arraysystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently.
- matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
- mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate.
- TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
Sources
- NVIDIA (2017). NVIDIA Tesla V100 GPU Architecture. Whitepaper.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world., CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI., FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., FP32FP32, abbr.: The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed., microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate., N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster., roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed., systolic arraysystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently., TensorRTTensorRT, prop. n.: NVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest., TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts.