How does information theory define negative entropy?

In information theory, entropy (Shannon entropy) measures the uncertainty, randomness, or unpredictability in a source or message. By contrast, negative entropy (negentropy) measures information, certainty, or predictability.

Shannon Entropy (H)

For a source with outcomes (symbols) having probabilities p_i, the Shannon entropy is:

H = −∑_i p_i · log₂(p_i)

• p_i: probability of the i-th outcome.
• log₂: logarithm base 2, so H is measured in bits.

High entropy: if outcomes are equally likely (e.g., fair coin: p_heads=0.5, p_tails=0.5), uncertainty H is maximal.

Low entropy: if one outcome is certain (e.g., biased coin that always lands heads: p_heads=1, p_tails=0), uncertainty H is zero.

Negentropy as Information

In this context, negentropy is the reduction of uncertainty—often identified with information.

Following Brillouin, information I can be expressed as the gap between the maximum possible entropy and the actual entropy:

I = H_max − H_actual

• H_max: maximum entropy under the same constraints.
• H_actual: observed/measured entropy of the source.

Example: English text has high negentropy because letters and words are not random (e.g., q is typically followed by u). This structure (redundancy) lowers entropy and increases information. A random string like “qzxvj” has high entropy and essentially zero negentropy (no meaningful information).