A wide range of machine learning applications such as privacy-preserving
learning, algorithmic fairness, and domain adaptation/generalization among
others, involve learning invariant representations of the data that aim to
achieve two competing goals: (a) maximize information or accuracy with respect
to a target response, and (b) maximize invariance or independence with respect
to a set of protected features (e.g., for fairness, privacy, etc). Despite
their wide applicability, theoretical understanding of the optimal tradeoffs --
with respect to accuracy, and invariance -- achievable by invariant
representations is still severely lacking. In this paper, we provide an
information theoretic analysis of such tradeoffs under both classification and
regression settings. More precisely, we provide a geometric characterization of
the accuracy and invariance achievable by any representation of the data; we
term this feasible region the information plane. We provide an inner bound for
this feasible region for the classification case, and an exact characterization
for the regression case, which allows us to either bound or exactly
characterize the Pareto optimal frontier between accuracy and invariance.
Although our contributions are mainly theoretical, a key practical application
of our results is in certifying the potential sub-optimality of any given
representation learning algorithm for either classification or regression
tasks. Our results shed new light on the fundamental interplay between accuracy
and invariance, and may be useful in guiding the design of future
representation learning algorithms.

本论文通过信息论的方法，对机器学习中不变表示的准确性和不变性之间的最优权衡进行了分析，并提出了一种几何特征，可以限定或精确刻画分类和回归任务的准确性和不变性之间的 Pareto 最优边界。

不变表示学习中的基本极限和权衡

Fundamental Limits and Tradeoffs in Invariant Representation Learning

Despite their great success, there is still no comprehensive theoretical
understanding of learning with Deep Neural Networks (DNNs) or their inner
organization. Previous work proposed to analyze DNNs in the \textit{Information
Plane}; i.e., the plane of the Mutual Information values that each layer
preserves on the input and output variables. They suggested that the goal of
the network is to optimize the Information Bottleneck (IB) tradeoff between
compression and prediction, successively, for each layer.
In this work we follow up on this idea and demonstrate the effectiveness of
the Information-Plane visualization of DNNs. Our main results are: (i) most of
the training epochs in standard DL are spent on {\emph compression} of the
input to efficient representation and not on fitting the training labels. (ii)
The representation compression phase begins when the training errors becomes
small and the Stochastic Gradient Decent (SGD) epochs change from a fast drift
to smaller training error into a stochastic relaxation, or random diffusion,
constrained by the training error value. (iii) The converged layers lie on or
very close to the Information Bottleneck (IB) theoretical bound, and the maps
from the input to any hidden layer and from this hidden layer to the output
satisfy the IB self-consistent equations. This generalization through noise
mechanism is unique to Deep Neural Networks and absent in one layer networks.
(iv) The training time is dramatically reduced when adding more hidden layers.
Thus the main advantage of the hidden layers is computational. This can be
explained by the reduced relaxation time, as this it scales super-linearly
(exponentially for simple diffusion) with the information compression from the
previous layer.

本研究通过信息平面可视化深度神经网络，发现在标准深度学习中，大部分训练周期都用于对输入进行压缩以生成有效表示，而非适应标签，当训练误差变小并且随机梯度下降下降进入随机扩散阶段时，表征压缩阶段开始。隐藏层的加入可以极大的缩短训练时间。