The performance of sentence encoders can be significantly improved through
the simple practice of fine-tuning using contrastive loss. A natural question
arises: what characteristics do models acquire during contrastive learning?
This paper theoretically and experimentally shows that contrastive-based
sentence encoders implicitly weight words based on information-theoretic
quantities; that is, more informative words receive greater weight, while
others receive less. The theory states that, in the lower bound of the optimal
value of the contrastive learning objective, the norm of word embedding
reflects the information gain associated with the distribution of surrounding
words. We also conduct comprehensive experiments using various models, multiple
datasets, two methods to measure the implicit weighting of models (Integrated
Gradients and SHAP), and two information-theoretic quantities (information gain
and self-information). The results provide empirical evidence that contrastive
fine-tuning emphasizes informative words.

通过对比损失的简单调优，句子编码器的性能可以得到显著提高。本文理论上和实验上展示出，在对比学习过程中模型获得了怎样的特征，即基于信息论的数量对词汇进行隐式加权。本文还使用多种模型、多个数据集、两种度量模型隐式加权的方法（集成渐变法和 SHAP）以及两个信息论数量（信息增益和自信息量）进行了全面实验。结果提供了对对比调优强调信息丰富词汇的经验证据。

基于对比学习的句子编码器隐式加权信息词汇

Contrastive Learning-based Sentence Encoders Implicitly Weight  Informative Words

Scenario-based question answering (SQA) requires retrieving and reading
paragraphs from a large corpus to answer a question which is contextualized by
a long scenario description. Since a scenario contains both keyphrases for
retrieval and much noise, retrieval for SQA is extremely difficult. Moreover,
it can hardly be supervised due to the lack of relevance labels of paragraphs
for SQA. To meet the challenge, in this paper we propose a joint
retriever-reader model called JEEVES where the retriever is implicitly
supervised only using QA labels via a novel word weighting mechanism. JEEVES
significantly outperforms a variety of strong baselines on multiple-choice
questions in three SQA datasets.

本文提出 JEEVES 这一基于联合检索 - 阅读器模型的算法作为一种自动问答技术。该算法使用了一种新颖的单词加权机制，通过 QA 标签隐式监督检索器，从而解决了当前 SQA 检索困难的问题。在三个 SQA 数据集上的多项选择问题对比实验证明，JEEVES 明显优于其他强基线算法。