With the exponential growth in data volume and the emergence of
data-intensive applications, particularly in the field of machine learning,
concerns related to resource utilization, privacy, and fairness have become
paramount. This paper focuses on the textual domain of data and addresses
challenges regarding encoding sentences to their optimized representations
through the lens of information-theory. In particular, we use empirical
estimates of mutual information, using the Donsker-Varadhan definition of
Kullback-Leibler divergence. Our approach leverages this estimation to train an
information-theoretic sentence embedding, called TexShape, for (task-based)
data compression or for filtering out sensitive information, enhancing privacy
and fairness. In this study, we employ a benchmark language model for initial
text representation, complemented by neural networks for information-theoretic
compression and mutual information estimations. Our experiments demonstrate
significant advancements in preserving maximal targeted information and minimal
sensitive information over adverse compression ratios, in terms of predictive
accuracy of downstream models that are trained using the compressed data.

通过信息理论的角度，本文致力于解决与将句子编码为其优化表示相关的挑战，特别关注数据领域中的资源利用、隐私和公平问题，并使用经验估计的相互信息，在 (task-based) 数据压缩或过滤敏感信息的过程中，训练了一种信息理论的句子嵌入模型，称为 TexShape。我们的实验证明，通过在压缩的数据上训练下游模型的预测准确性来保留最大化的目标信息和最小化的敏感信息，可以在不良的压缩比例方面取得显著进展。

TexShape: 语言模型的信息论句子嵌入

TexShape: Information Theoretic Sentence Embedding for Language Models

We review the current literature concerned with information plane analyses of
neural network classifiers. While the underlying information bottleneck theory
and the claim that information-theoretic compression is causally linked to
generalization are plausible, empirical evidence was found to be both
supporting and conflicting. We review this evidence together with a detailed
analysis of how the respective information quantities were estimated. Our
survey suggests that compression visualized in information planes is not
necessarily information-theoretic, but is rather often compatible with
geometric compression of the latent representations. This insight gives the
information plane a renewed justification.
Aside from this, we shed light on the problem of estimating mutual
information in deterministic neural networks and its consequences.
Specifically, we argue that even in feed-forward neural networks the data
processing inequality need not hold for estimates of mutual information.
Similarly, while a fitting phase, in which the mutual information between the
latent representation and the target increases, is necessary (but not
sufficient) for good classification performance, depending on the specifics of
mutual information estimation such a fitting phase need not be visible in the
information plane.

本文就信息平面分析、神经网络分类器、信息论压缩、潜在表示、互信息估计等方面的问题进行了综述和分析，研究发现信息压缩不一定与信息论有关，而是与潜在表示的几何压缩相容。此外还探讨了神经网络互信息估计的问题及其后果。