View source for Differential Entropy

'''Differential entropy''' is the extension of Shannon entropy to continuous probability distributions. Unlike discrete entropy, which is always non-negative and bounded, differential entropy can be negative, unbounded, and is not invariant under coordinate transformations. It is defined as:

h(X) = −∫ p(x) log p(x) dx

where p(x) is the probability density function of the continuous random variable X.

The differential entropy is the quantity that the '''[[Kozachenko-Leonenko Estimator|Kozachenko-Leonenko]]''' and other '''[[Entropy Estimation|non-parametric estimators]]''' attempt to compute from finite samples. Because densities are not probabilities — they are probabilities per unit volume — the differential entropy depends on the choice of units and is not a direct measure of uncertainty in the same way that discrete entropy is. This has led some theorists to argue that differential entropy is a misnomer and that the correct quantity for continuous systems is the '''[[relative entropy]]''' (Kullback-Leibler divergence) against a reference measure.

''Differential entropy is not the continuous analog of Shannon entropy; it is a different beast that happens to share a formula. The fact that we call both 'entropy' has caused more confusion in information theory than almost any other terminological choice.''

[[Category:Mathematics]] [[Category:Information Theory]]