Skip to content
Last Dictionary
Plane
Mathematics & StatisticsData & DatasetsVision, Speech & Perception

Jaccard similarity

nounAlso: Jaccard index, Jaccard coefficient, Jaccard distance, Tanimoto coefficient

In plain wordsA simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0.

  1. A measure of overlap between two finite sets, equal to the size of their intersection divided by the size of their union. It ranges from 0 (disjoint) to 1 (identical), and one minus it, the Jaccard distance, is a true metric. Applied to documents represented as sets of n-grams ('shingles'), it quantifies textual near-duplication and is the quantity that MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints. estimates; deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online. pipelines commonly treat pairs above a threshold of roughly 0.7–0.85 as duplicates. The same ratio computed over pixels or boxes is intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. in object detection and segmentation; weighted and binary-vector forms are used in recommender systems and in chemistry, where it is called the Tanimoto coefficient.

    The two scraped pages had a Jaccard similarity of 0.92 over 5-gram shingles, so one was dropped.

J(A,B)=∣A∩B∣∣A∪B∣J(A,B) = \frac{|A \cap B|}{|A \cup B|}

Origin

Named after the Swiss botanist Paul Jaccard, who introduced it in 1901 as a 'coefficient de communauté' (coefficient of community) for comparing the plant species of different Alpine districts; his 1912 English-language paper in New Phytologist spread it. The same ratio had appeared in 1884 as the American geologist Grove Karl Gilbert's 'ratio of verification' for scoring tornado forecasts, and Taffee Tanimoto reformulated it independently at IBM in 1958.

Usage note

In weather-forecast verification, where Gilbert's version arose, the same quantity is called the critical success index or threat score.

Related

See also
  • MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints.
  • deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online.
  • intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
  • locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared.
  • recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
Contrast with
  • cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.

Sources

  1. Jaccard, P. (1901). Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin de la Société Vaudoise des Sciences Naturelles 37.
  2. Jaccard, P. (1912). The Distribution of the Flora in the Alpine Zone. New Phytologist 11(2).

Mentioned in

deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online., locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared., MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints.