Skip to content
Last Dictionary
Plane
Mathematics & StatisticsDeep LearningInterpretability & Steering

information bottleneck

nounAlso: information bottleneck principle, information bottleneck method, IB, variational information bottleneck

In plain wordsThe idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does.

  1. 1

    A principle and method, introduced by Naftali Tishby, Fernando Pereira and William Bialek in 1999, for summarizing an input X by a representation T that discards as much of X as it can while staying informative about a target Y: one minimizes the mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated. I(X; T) minus β times I(T; Y), with β setting the balance between compression and prediction. Unlike classic rate–distortion theory, which needs a distortion measure chosen in advance, it lets the target variable define what counts as relevant. The variational information bottleneck (Alexander Alemi and colleagues, 2016) made the objective trainable as a regularizer for neural networks, closely related to the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. of VAEsvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples..

    Treat the summary as an information bottleneck: keep only what helps predict the label.

  2. 2

    Deep learning theorycontested

    The hypothesis, advanced by Naftali Tishby and colleagues in 2015–2017, that deep-network training proceeds in two phases — first fitting the labels, then compressing representations of the input — and that this compression explains generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.. Later work (Andrew Saxe and colleagues, 2018) found that apparent compression depends on the activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. and on how mutual information is estimated, and the claim remains disputed.

min⁡p(t∣x)  I(X;T)−β I(T;Y)\min_{p(t\mid x)} \; I(X;T) - \beta\, I(T;Y)

Origin

Introduced by Naftali Tishby, Fernando Pereira and William Bialek in 'The Information Bottleneck Method' (Allerton Conference, 1999; arXiv 2000). Tishby and Noga Zaslavsky proposed it as a theory of deep learning in 2015, and Ravid Shwartz-Ziv and Tishby's 2017 'Opening the Black Box of Deep Neural Networks via Information' popularized the compression-phase claim.

Related

See also
  • mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
  • information theoryinformation theory, n.: A branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent.
  • minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.
  • generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
  • representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
  • variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.

Sources

  1. Tishby, N., Pereira, F. C. & Bialek, W. (1999). The Information Bottleneck Method. Proc. 37th Allerton Conference on Communication, Control and Computing.
  2. Saxe, A. M. et al. (2018). On the Information Bottleneck Theory of Deep Learning. ICLR 2018.

Mentioned in

information theoryinformation theory, n.: A branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent., mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.