Embedders play a central role in machine learning, projecting any object into
numerical representations that can, in turn, be leveraged to perform various
downstream tasks. The evaluation of embedding models typically depends on
domain-specific empirical approaches utilizing downstream tasks, primarily
because of the lack of a standardized framework for comparison. However,
acquiring adequately large and representative datasets for conducting these
assessments is not always viable and can prove to be prohibitively expensive
and time-consuming. In this paper, we present a unified approach to evaluate
embedders. First, we establish theoretical foundations for comparing embedding
models, drawing upon the concepts of sufficiency and informativeness. We then
leverage these concepts to devise a tractable comparison criterion (information
sufficiency), leading to a task-agnostic and self-supervised ranking procedure.
We demonstrate experimentally that our approach aligns closely with the
capability of embedding models to facilitate various downstream tasks in both
natural language processing and molecular biology. This effectively offers
practitioners a valuable tool for prioritizing model trials.

本研究提出了一种统一的评估嵌入模型的方法，通过建立理论基础并利用信息充分性的概念进行比较，为自动排序过程提供了一种可操作的标准，展示了实验证明该方法与嵌入模型在自然语言处理和分子生物学中支持各种下游任务的能力密切相关，为实践者提供了一种有价值的模型试验优先级工具。

嵌入模型何时比其他模型更有前途？

When is an Embedding Model More Promising than Another?

We present new results to model and understand the role of encoder-decoder
design in machine learning (ML) from an information-theoretic angle. We use two
main information concepts, information sufficiency (IS) and mutual information
loss (MIL), to represent predictive structures in machine learning. Our first
main result provides a functional expression that characterizes the class of
probabilistic models consistent with an IS encoder-decoder latent predictive
structure. This result formally justifies the encoder-decoder forward stages
many modern ML architectures adopt to learn latent (compressed) representations
for classification. To illustrate IS as a realistic and relevant model
assumption, we revisit some known ML concepts and present some interesting new
examples: invariant, robust, sparse, and digital models. Furthermore, our IS
characterization allows us to tackle the fundamental question of how much
performance (predictive expressiveness) could be lost, using the cross entropy
risk, when a given encoder-decoder architecture is adopted in a learning
setting. Here, our second main result shows that a mutual information loss
quantifies the lack of expressiveness attributed to the choice of a (biased)
encoder-decoder ML design. Finally, we address the problem of universal
cross-entropy learning with an encoder-decoder design where necessary and
sufficiency conditions are established to meet this requirement. In all these
results, Shannon's information measures offer new interpretations and
explanations for representation learning.

我们从信息论的角度提出了有关编码器 - 解码器设计在机器学习中发挥作用的新结果，使用信息充分性和互信息损失来表示机器学习中的预测结构，并使用香农的信息度量为表示学习提供了新的解释。