This paper proposes an information-theoretic representation learning
framework, named conditional information flow maximization, to extract
noise-invariant sufficient representations for the input data and target task.
It promotes the learned representations have good feature uniformity and
sufficient predictive ability, which can enhance the generalization of
pre-trained language models (PLMs) for the target task. Firstly, an information
flow maximization principle is proposed to learn more sufficient
representations by simultaneously maximizing both input-representation and
representation-label mutual information. In contrast to information bottleneck,
we handle the input-representation information in an opposite way to avoid the
over-compression issue of latent representations. Besides, to mitigate the
negative effect of potential redundant features, a conditional information
minimization principle is designed to eliminate negative redundant features
while preserve noise-invariant features from the input. Experiments on 13
language understanding benchmarks demonstrate that our method effectively
improves the performance of PLMs for classification and regression. Extensive
experiments show that the learned representations are more sufficient, robust
and transferable.

本论文提出了一个名为条件信息流最大化的信息论表示学习框架，用于提取输入数据和目标任务的噪声不变的充分表示。该方法通过最大化输入 - 表示和表示 - 标签互信息，同时促进学习到的表示具有较好的特征一致性和充分的预测能力，从而增强了预训练语言模型在目标任务中的泛化能力。实验证明，该方法有效地提升了用于分类和回归的预训练语言模型的性能，并且学到的表示更加充分、鲁棒和可迁移。