Skip to content
Last Dictionary
Plane
Mathematics & StatisticsAgents & HarnessesLanguage Models & Transformers

cosine similarity

nounAlso: cosine distance, cosine score, cosine measure

In plain wordsA score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.

  1. A similarity score for two nonzero vectorsvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points. that compares only their directions. Each vector is first divided by its Euclidean normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes. and the two unit vectors are then combined by a dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles., so the score equals the cosine of the angle between them: 1 for vectors pointing the same way, 0 for orthogonal ones, −1 for opposite ones. Discarding magnitude suits data in which size reflects something incidental, such as how long a document is. It is the default metric for comparing text embeddingstext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning. in semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'., retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. and vector databasesvector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., and for scoring image–text pairs in CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.-style contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.. For unit-length vectors it equals the dot product, and squared Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length. equals 2 − 2 × cosine similarity, so all three give the same ranking. Cosine distance, one minus the similarity, is not a true metric because it can violate the triangle inequality.

    Anything below 0.75 cosine similarity to the query is dropped before reranking.

cos⁡(a,b)=a⋅b∥a∥ ∥b∥\cos(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}

Origin

The angle between vectors is classical geometry, but the cosine became a standard similarity score in information retrieval through Gerard Salton's SMART retrieval system, developed from the 1960s, and was codified in the vector space model of Salton, Andrew Wong and Chung-Shu Yang (1975). It passed to word embeddings such as word2vec (2013) and then to modern embedding search.

Usage note

Not the same as Pearson correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other., which subtracts each vector's mean first. Embeddings from one model often share a common direction that inflates every similarity, so scores are comparable only within a single embedding model.

Related

See also
  • dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles.
  • Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.
  • text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
  • vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's.
  • semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'.
  • normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.
  • embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
Contrast with
  • Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.

Sources

  1. Salton, G., Wong, A. & Yang, C. S. (1975). A Vector Space Model for Automatic Indexing. Communications of the ACM, 18(11).

Mentioned in

embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., content-based filteringcontent-based filtering, n.: Recommending items that resemble what you already liked, judged by the items' own features, such as a film's genre, cast and plot, rather than by other people's choices., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles., Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length., k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words., speaker verificationspeaker verification, n.: Checking whether a voice really belongs to the person it claims to be, as when a bank confirms your identity by the sound of your voice on the phone., text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning., TF-IDFTF-IDF, abbr.: A simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it., two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison., vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's., word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text., CLIP scoreCLIP score, n.: A score for how well a picture matches a text description, computed by an AI model that was trained to pair images with their captions., GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text., Jaccard similarityJaccard similarity, n.: A simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared.