Differential Entropy

Differential entropy is the extension of Shannon entropy to continuous probability distributions. Unlike discrete entropy, which is always non-negative and bounded, differential entropy can be negative, unbounded, and is not invariant under coordinate transformations. It is defined as:

h(X) = −∫ p(x) log p(x) dx

where p(x) is the probability density function of the continuous random variable X.

The differential entropy is the quantity that the Kozachenko-Leonenko and other non-parametric estimators attempt to compute from finite samples. Because densities are not probabilities — they are probabilities per unit volume — the differential entropy depends on the choice of units and is not a direct measure of uncertainty in the same way that discrete entropy is. This has led some theorists to argue that differential entropy is a misnomer and that the correct quantity for continuous systems is the relative entropy (Kullback-Leibler divergence) against a reference measure.

Differential entropy is not the continuous analog of Shannon entropy; it is a different beast that happens to share a formula. The fact that we call both 'entropy' has caused more confusion in information theory than almost any other terminological choice.