Learned representations of scientific documents can serve as valuable input
features for downstream tasks, without the need for further fine-tuning.
However, existing benchmarks for evaluating these representations fail to
capture the diversity of relevant tasks. In response, we introduce SciRepEval,
the first comprehensive benchmark for training and evaluating scientific
document representations. It includes 25 challenging and realistic tasks, 11 of
which are new, across four formats: classification, regression, ranking and
search. We then use the benchmark to study and improve the generalization
ability of scientific document representation models. We show how
state-of-the-art models struggle to generalize across task formats, and that
simple multi-task training fails to improve them. However, a new approach that
learns multiple embeddings per document, each tailored to a different format,
can improve performance. We experiment with task-format-specific control codes
and adapters in a multi-task setting and find that they outperform the existing
single-embedding state-of-the-art by up to 1.5 points absolute.

本文介绍了 SciRepEval，第一个全面评估科学文档表示方法的基准测试，可用于分类、回归、排名和搜索等多项任务，研究表明最新模型普遍存在跨任务格式泛化能力差的问题，但学习每个文档的多个嵌入以针对不同格式得到改善，实验发现多任务设置下的任务格式特定控制码和适配器优于现有的最新单嵌入模型。

SciRepEval：科学文档表示的多格式基准

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

Learning scientific document representations can be substantially improved
through contrastive learning objectives, where the challenge lies in creating
positive and negative training samples that encode the desired similarity
semantics. Prior work relies on discrete citation relations to generate
contrast samples. However, discrete citations enforce a hard cut-off to
similarity. This is counter-intuitive to similarity-based learning, and ignores
that scientific papers can be very similar despite lacking a direct citation -
a core problem of finding related research. Instead, we use controlled nearest
neighbor sampling over citation graph embeddings for contrastive learning. This
control allows us to learn continuous similarity, to sample hard-to-learn
negatives and positives, and also to avoid collisions between negative and
positive samples by controlling the sampling margin between them. The resulting
method SciNCL outperforms the state-of-the-art on the SciDocs benchmark.
Furthermore, we demonstrate that it can train (or tune) models
sample-efficiently, and that it can be combined with recent training-efficient
methods. Perhaps surprisingly, even training a general-domain language model
this way outperforms baselines pretrained in-domain.

本文提出了一种通过对比学习来改善科学文档表示的方法 —— 使用引文图嵌入进行所控制的最近邻抽样来学习连续的相似度，内容主要涉及对比学习、科学文档表示、引文图嵌入、相似度语义以及相关研究。