The Information Bottleneck principle offers both a mechanism to explain how
deep neural networks train and generalize, as well as a regularized objective
with which to train models. However, multiple competing objectives are proposed
in the literature, and the information-theoretic quantities used in these
objectives are difficult to compute for large deep neural networks, which in
turn limits their use as a training objective. In this work, we review these
quantities and compare and unify previously proposed objectives, which allows
us to develop surrogate objectives more friendly to optimization without
relying on cumbersome tools such as density estimation. We find that these
surrogate objectives allow us to apply the information bottleneck to modern
neural network architectures. We demonstrate our insights on MNIST, CIFAR-10
and Imagenette with modern DNN architectures (ResNets).

本研究旨在比较和统一先前提出的信息瓶颈原则中的多个竞争目标，并开发更友好的替代目标，以便将信息瓶颈应用于现代深度神经网络结构，无需依赖于像密度估计这样的繁琐工具。研究结果可以在现代 DNN 结构（ResNets）上展示。

深度学习中的信息瓶颈解析：统一信息理论目标

Unpacking Information Bottlenecks: Unifying Information-Theoretic  Objectives in Deep Learning

The goal in extreme multi-label classification is to learn a classifier which
can assign a small subset of relevant labels to an instance from an extremely
large set of target labels. Datasets in extreme classification exhibit a long
tail of labels which have small number of positive training instances. In this
work, we pose the learning task in extreme classification with large number of
tail-labels as learning in the presence of adversarial perturbations. This view
motivates a robust optimization framework and equivalence to a corresponding
regularized objective.
Under the proposed robustness framework, we demonstrate efficacy of Hamming
loss for tail-label detection in extreme classification. The equivalent
regularized objective, in combination with proximal gradient based
optimization, performs better than state-of-the-art methods on propensity
scored versions of precision@k and nDCG@k(upto 20% relative improvement over
PFastreXML - a leading tree-based approach and 60% relative improvement over
SLEEC - a leading label-embedding approach). Furthermore, we also highlight the
sub-optimality of a sparse solver in a widely used package for large-scale
linear classification, which is interesting in its own right. We also
investigate the spectral properties of label graphs for providing novel
insights towards understanding the conditions governing the performance of
Hamming loss based one-vs-rest scheme vis-\`a-vis label embedding methods.

本文探讨了在存在对抗扰动的情况下的极端分类中的学习任务，并提出了一种鲁棒优化框架和相应的正则化目标，发现 Hamming 损失可作为尾标签检测的有效方法。通过结合正则化目标和基于 Proximal 梯度的优化方法，相对于 PFastreXML 和 SLEEC 等现有方法在 precision@k 和 nDCG@k 上提供了优异的性能。同时，我们还研究了标签图的谱特性，提供了新的洞察力以了解 Hamming 损失的一对多方法和标签嵌入方法之间性能的条件限制。