Hierarchical Clustering

Hierarchical clustering is a family of algorithms that build a nested sequence of partitions, represented as a tree structure called a dendrogram. Unlike k-means, which requires prespecifying the number of clusters, hierarchical clustering defers that choice to a later stage: the analyst decides where to cut the tree. This apparent flexibility is a trap — it replaces the hard problem of choosing k with the equally hard problem of choosing a cutoff, and the cutoff choice is not independent of the linkage criterion used to build the tree.

The algorithm proceeds either agglomeratively (bottom-up, merging the closest pair of clusters at each step) or divisively (top-down, splitting the most heterogeneous cluster). The "closest pair" is defined by a linkage criterion: single linkage (minimum inter-cluster distance), complete linkage (maximum distance), average linkage (mean distance), or Ward's method (minimum variance increase). Each criterion produces a different dendrogram from the same data, and the choice between them is rarely justified by the data's geometry. Single linkage excels at finding elongated clusters but is sensitive to noise; complete linkage favors compact, spherical clusters; Ward's method assumes clusters are convex and of similar size. The algorithm does not discover hierarchy; it imposes one.

The dendrogram is often treated as a phylogenetic or evolutionary tree, especially in biological applications. This is a dangerous analogy. A phylogenetic tree represents historical divergence; a dendrogram represents algorithmic merging order. These are not the same thing, and interpreting a dendrogram as history commits the same error as interpreting a principal component as a causal factor.

Hierarchical clustering promises to reveal the deep structure of data, but what it reveals is the structure of its own assumptions about what "deep" means. A dendrogram is not a map of reality; it is a trace of the algorithm's journey through similarity space, and the journey is determined by the destination it was programmed to reach.