In this paper, we use matrix information theory as an analytical tool to
analyze the dynamics of the information interplay between data representations
and classification head vectors in the supervised learning process.
Specifically, inspired by the theory of Neural Collapse, we introduce matrix
mutual information ratio (MIR) and matrix entropy difference ratio (HDR) to
assess the interactions of data representation and class classification heads
in supervised learning, and we determine the theoretical optimal values for MIR
and HDR when Neural Collapse happens. Our experiments show that MIR and HDR can
effectively explain many phenomena occurring in neural networks, for example,
the standard supervised training dynamics, linear mode connectivity, and the
performance of label smoothing and pruning. Additionally, we use MIR and HDR to
gain insights into the dynamics of grokking, which is an intriguing phenomenon
observed in supervised training, where the model demonstrates generalization
capabilities long after it has learned to fit the training data. Furthermore,
we introduce MIR and HDR as loss terms in supervised and semi-supervised
learning to optimize the information interactions among samples and
classification heads. The empirical results provide evidence of the method's
effectiveness, demonstrating that the utilization of MIR and HDR not only aids
in comprehending the dynamics throughout the training process but can also
enhances the training procedure itself.

利用矩阵信息理论作为分析工具，本文研究了数据表示和分类头向量在监督学习过程中的信息交互动态，引入了矩阵互信息比率（MIR）和矩阵熵差比率（HDR）来评估数据表示和分类头在监督学习中的相互作用，并确定了当发生神经坍缩时 MIR 和 HDR 的理论最优值。实验结果表明，MIR 和 HDR 可以有效解释神经网络中发生的许多现象，例如标准监督训练动态、线性模式连通性以及标签平滑和修剪的性能。此外，将 MIR 和 HDR 引入监督和半监督学习中作为损失项，以优化样本和分类头之间的信息交互。实证结果证明了该方法的有效性，不仅有助于理解训练过程中的动态，还能增强训练过程本身。

揭示督导学习中信息互动的动态

Unveiling the Dynamics of Information Interplay in Supervised Learning

In this paper, we provide a comprehensive toolbox for understanding and
enhancing self-supervised learning (SSL) methods through the lens of matrix
information theory. Specifically, by leveraging the principles of matrix mutual
information and joint entropy, we offer a unified analysis for both contrastive
and feature decorrelation based methods. Furthermore, we propose the matrix
variational masked auto-encoder (M-MAE) method, grounded in matrix information
theory, as an enhancement to masked image modeling. The empirical evaluations
underscore the effectiveness of M-MAE compared with the state-of-the-art
methods, including a 3.9% improvement in linear probing ViT-Base, and a 1%
improvement in fine-tuning ViT-Large, both on ImageNet.

本文通过矩阵信息理论的视角提供了一个全面的工具箱，用于理解和增强自监督学习方法，其中包括对比方法、特征去相关方法的统一分析，并提出了基于矩阵信息理论的矩阵变分掩膜自编码器 (M-MAE) 方法作为对掩膜图像建模的一种增强方法。经验证实，M-MAE 相对于最先进方法，在 ImageNet 上线性探测 ViT-Base 性能提升了 3.9%，微调 ViT-Large 性能提升了 1%。