silhouette score
In plain wordsA number between −1 and 1 that grades a grouping of data by checking, for each point, whether it sits closer to its own group than to the next-nearest group.
An internal clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size. validity index that needs no ground-truth labels. For each point i, a(i) is its mean distance to the other members of its own cluster and b(i) is its mean distance to the members of the nearest other cluster; the point's silhouette is (b − a) divided by the larger of a and b. Values near +1 mean the point sits well inside its cluster, values near 0 that it lies on a boundary, and negative values that it would sit more comfortably in the neighboring cluster. Averaging over all points gives the silhouette score of the whole clustering, and per-cluster silhouette plots reveal weak clusters. A common use is choosing the number of clusters by maximizing the mean silhouette. Exact computation needs all pairwise distances, costing O(n²) time and memory, so sampled or centroid-based simplifications are used for large datasets. The index favors compact, convex, well-separated clusters and can mislead for elongated or density-based ones.
The mean silhouette peaked at k = 4, so we went with four customer segments.
Origin
Introduced by the Belgian statistician Peter Rousseeuw in 'Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis' (Journal of Computational and Applied Mathematics, 1987). Leonard Kaufman and Rousseeuw called the maximum mean silhouette over candidate clusterings the 'silhouette coefficient' in their book Finding Groups in Data (1990).
Usage note
Like other internal indices (Davies–Bouldin, Calinski–Harabasz), it measures geometric tidiness, not whether the clusters are meaningful for the task at hand.
Related
- clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
- k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.
- model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
- hierarchical clusteringhierarchical clustering, n.: A way of grouping data that builds a family tree of clusters, joining the most similar items step by step, so you can cut the tree at whatever level of detail you need.
- Euclidean distanceEuclidean distance, n.: How far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.
Sources
- Rousseeuw, P. J. (1987). Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis. Journal of Computational and Applied Mathematics 20: 53–65.
- Kaufman, L. & Rousseeuw, P. J. (1990). Finding Groups in Data: An Introduction to Cluster Analysis. New York: Wiley.
Mentioned in
clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., hierarchical clusteringhierarchical clustering, n.: A way of grouping data that builds a family tree of clusters, joining the most similar items step by step, so you can cut the tree at whatever level of detail you need., k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.