dot product
In plain wordsA score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles.
An operation that scores how closely two same-dimension vectorsvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points. line up, weighted by their sizes. From coordinates, it pairs corresponding components, multiplies each pair and sums the products; read geometrically, the same number is the product of the two lengths (normsnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.) and the cosine of the angle between them, so it is positive for vectors pointing roughly the same way, zero for orthogonal ones and negative for vectors pointing apart. The dot product is the basic act of neural computation: each artificial neuronartificial neuron, n.: The basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output. takes a dot product of its weights with its inputs, every entry of a matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing. is one, and scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information. scores how well each query matches each key by their dot product divided by the square root of the key dimension. Dividing by both norms gives cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.. The general notion, an inner product, extends the idea to other vector spaces, including spaces of functions.
Because the embeddings are unit-normalized, ranking by dot product and by cosine similarity gives the same order.
Origin
The operation is implicit in William Rowan Hamilton's quaternion product and Hermann Grassmann's inner product of the 1840s–1860s. The term 'scalar product' is recorded in William Kingdon Clifford's Elements of Dynamic (1878), and the name 'dot product', from the notation a · b, appears in Edwin Bidwell Wilson's 1901 textbook Vector Analysis, based on Josiah Willard Gibbs's lectures.
Related
- cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
- vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points.
- matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
- scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
- normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.
- orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing., scalarscalar, n.: A single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars., scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information., text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning., two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison., vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.