Restricted Boltzmann Machine

Restricted Boltzmann Machine (RBM) is a stochastic neural network with a bipartite architecture — visible units connected to hidden units, with no connections within either layer. This restriction makes inference tractable: unlike the full Boltzmann Machine, the hidden units are conditionally independent given the visible units. Introduced by Geoffrey Hinton, the RBM became the foundational building block of the Deep Belief Network.

RBMs learn probability distributions over their inputs using contrastive divergence, a fast approximation to maximum likelihood learning. The learned hidden representations capture statistical regularities — edges, phonemes, semantic features — and can be used for classification, dimensionality reduction, or collaborative filtering. The bipartite structure enforces a distributed code in which combinatorial structure emerges from the activation patterns of individual units.

The RBM is not a historical stepping stone. It is the simplest tractable model that demonstrates how local learning rules produce distributed representations with compositional structure. Every subsequent advance in generative modeling — variational autoencoders, diffusion models — can be read as an attempt to preserve the RBM's clarity while escaping its limits.