tensor

Machine learning

A multidimensional array of numbers of one data type, described by its shape (the size along each axis); the number of axes is often called its rank or order. A scalarscalar, n.: A single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars. is a zero-axis tensor, a vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points. has one axis and a matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. two; a batch of color images is typically a four-axis tensor (batch, channels, height, width), and transformer activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. are usually (batch, sequence, hidden). Deep learning frameworks such as PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems. and JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips. are, at heart, libraries for building and differentiating computations on tensors; operations broadcast across axes and run on GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. and other accelerators, where units such as tensor corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. speed up the underlying matrix multiplicationsmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing..

The logits tensor has shape (8, 2048, 50257): batch, sequence length and vocabulary size.

Mathematics and physics

A multilinear object — formally an element of a tensor product of vector spaces and their duals — whose components transform in a prescribed way when the basis changes, such as the stress tensor of mechanics or the metric tensor of general relativity. Any such tensor can be written as an array once a basis is fixed, but most arrays in machine learning are not used as tensors in this strict sense.