MINE Framework

The MINE framework (Mutual Information Neural Estimation) is a neural network-based approach to estimating mutual information from samples, developed by Belghazi et al. in 2018. Unlike binning, kernel density estimation, or the KSG estimator, MINE does not rely on nearest-neighbor distances or density estimates. Instead, it trains a neural network to maximize a lower bound on mutual information derived from the Donsker-Varadhan representation of the Kullback-Leibler divergence. The network learns a function that discriminates between samples drawn from the joint distribution and samples drawn from the product of marginals.

MINE is particularly valuable in high-dimensional regimes where traditional non-parametric estimators fail due to the curse of dimensionality. In image analysis, representation learning, and reinforcement learning, the dimensions may number in the thousands or millions, making KSG and binning infeasible. MINE scales gracefully with dimension because the neural network learns a compressed representation in which the mutual information is easier to estimate. The cost of this scalability is epistemic opacity: unlike KSG, whose estimate can be traced to specific nearest-neighbor distances, MINE produces a number whose relationship to the true mutual information is mediated by an uninterpretable neural network.

MINE represents a fundamental trade-off in modern machine learning: we can either understand what we are measuring or measure high-dimensional dependencies accurately, but we cannot do both. The framework's popularity is not a sign that neural estimation is superior to classical methods; it is a sign that the problems we care about have outgrown the assumptions that made classical methods trustworthy.