The Information Bottleneck (IB) provides an information theoretic principle
for representation learning, by retaining all information relevant for
predicting label while minimizing the redundancy. Though IB principle has been
applied to a wide range of applications, its optimization remains a challenging
problem which heavily relies on the accurate estimation of mutual information.
In this paper, we present a new strategy, Variational Self-Distillation (VSD),
which provides a scalable, flexible and analytic solution to essentially
fitting the mutual information but without explicitly estimating it. Under
rigorously theoretical guarantee, VSD enables the IB to grasp the intrinsic
correlation between representation and label for supervised training.
Furthermore, by extending VSD to multi-view learning, we introduce two other
strategies, Variational Cross-Distillation (VCD) and Variational
Mutual-Learning (VML), which significantly improve the robustness of
representation to view-changes by eliminating view-specific and task-irrelevant
information. To verify our theoretically grounded strategies, we apply our
approaches to cross-modal person Re-ID, and conduct extensive experiments,
where the superior performance against state-of-the-art methods are
demonstrated. Our intriguing findings highlight the need to rethink the way to
estimate mutual

本篇文章提出了一种新策略，基于变分自我蒸馏，为信息瓶颈提供了一种可扩展，灵活和分析的解决方案，以便实现对监督训练中表示与标签之间内在相关性的掌握；并且通过将 VSD 扩展到多视图学习，引入了另外两个策略，即变分交叉蒸馏和变分相互学习，这显著提高了表示对视图变化的鲁棒性。