Hierarchical Clustering

Hierarchical clustering is a family of algorithms that build a nested sequence of partitions, represented as a tree structure called a dendrogram. Unlike k-means, which requires prespecifying the number of clusters, hierarchical clustering defers that choice to a later stage: the analyst decides where to cut the tree. This apparent flexibility is a trap — it replaces the hard problem of choosing k with the equally hard problem of choosing a cutoff, and the cutoff choice is not independent of the linkage criterion used to build the tree.

The algorithm proceeds either agglomeratively (bottom-up, merging the closest pair of clusters at each step) or divisively (top-down, splitting the most heterogeneous cluster). The "closest pair" is defined by a linkage criterion: single linkage (minimum inter-cluster distance), complete linkage (maximum distance), average linkage (mean distance), or Ward's method (minimum variance increase). Each criterion produces a different dendrogram from the same data, and the choice between them is rarely justified by the data's geometry. Single linkage excels at finding elongated clusters but is sensitive to noise; complete linkage favors compact, spherical clusters; Ward's method assumes clusters are convex and of similar size. The algorithm does not discover hierarchy; it imposes one.

The dendrogram is often treated as a phylogenetic or evolutionary tree, especially in biological applications. This is a dangerous analogy. A phylogenetic tree represents historical divergence; a dendrogram represents algorithmic merging order. These are not the same thing, and interpreting a dendrogram as history commits the same error as interpreting a principal component as a causal factor.

Hierarchical clustering promises to reveal the deep structure of data, but what it reveals is the structure of its own assumptions about what "deep" means. A dendrogram is not a map of reality; it is a trace of the algorithm's journey through similarity space, and the journey is determined by the destination it was programmed to reach.\n== The Algorithm-Reality Boundary ==\n\nThe article's closing claim — that a dendrogram is not a map of reality but a trace of the algorithm's journey — is correct but incomplete. It treats the dendrogram as purely constructed, ignoring an important asymmetry: some datasets genuinely possess hierarchical structure, and the algorithm's job is to recover it.\n\nThis is not a minor point. In phylogenetics, the evolutionary tree is not a metaphor for the dendrogram; the dendrogram is an attempt to estimate the tree. The distinction is epistemological, not merely terminological. A phylogenetic tree represents a historical process (speciation and divergence) that produced hierarchical structure in the genetic data. The dendrogram is an inference about that structure, not an imposition of it. When the algorithm fails — when it groups species that did not share a recent common ancestor — it is making an error about the world, not merely revealing its own assumptions.\n\nThe same applies to non-biological domains. In taxonomy, linguistics, and even some social network analyses, hierarchical structure exists independently of the clustering algorithm. The question is not whether hierarchy is imposed but whether the algorithm's imposed hierarchy corresponds to the data's natural hierarchy. This is an empirical question about the data-generating process, not a philosophical question about the nature of representation.\n\nThe deeper systems-theoretic point is that the algorithm-reality boundary is not a binary opposition but a continuum. At one extreme, the data has no hierarchical structure and the dendrogram is pure artifact. At the other extreme, the data has strong hierarchical structure and the dendrogram is a reasonable estimate. Most real datasets fall somewhere in between, and the analyst's job is to determine where — not to declare that all dendrograms are fiction.\n\nThe framework of model selection provides the appropriate language: hierarchical clustering is a model, the dendrogram is a model output, and the question is whether the model is appropriate to the data-generating process. A model that is wrong about the world is not merely telling us about itself. It is making a claim that can be tested against external evidence.\n\nThe claim that dendrograms are never maps of reality is as one-sided as the claim that they always are. The truth is situated: it depends on whether the data-generating process produced hierarchical structure, and whether the linkage criterion is appropriate to that structure. The algorithm does not discover hierarchy in a vacuum, but it does not merely impose it in a vacuum either.