information theory

The mathematical theory, founded on probability, of how much uncertainty a message resolves and of the limits this places on coding: how compactly the output of a source can be represented and how fast data can cross a noisy channel without error. Its central quantities — entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy. (average uncertainty), cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. and mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated. — are measured in bits (base-2 logarithms) or nats (natural logarithms). Its main theorems set hard limits: the source coding theorem says data cannot be losslessly compressed below its entropy, and the noisy-channel coding theorem says reliable communication is possible up to, but not beyond, a channel's capacity. Information theory supplies much of machine learning's vocabulary and objectives: language models are trained to minimize cross-entropy, so a better predictor is a better compressor; perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. and bits per byte measure modeling quality; contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things. maximizes bounds on mutual information; and minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress., the information bottleneckinformation bottleneck, n.: The idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does. and rate–distortion theory frame learning as compression.

In information-theory terms, a model with lower cross-entropy is literally a better text compressor.