Learned representations of scientific documents can serve as valuable input
features for downstream tasks, without the need for further fine-tuning.
However, existing benchmarks for evaluating these representations fail to
capture the diversity of relevant tasks. In response, we introduce SciRepEval,
the first comprehensive benchmark for training and evaluating scientific
document representations. It includes 25 challenging and realistic tasks, 11 of
which are new, across four formats: classification, regression, ranking and
search. We then use the benchmark to study and improve the generalization
ability of scientific document representation models. We show how
state-of-the-art models struggle to generalize across task formats, and that
simple multi-task training fails to improve them. However, a new approach that
learns multiple embeddings per document, each tailored to a different format,
can improve performance. We experiment with task-format-specific control codes
and adapters in a multi-task setting and find that they outperform the existing
single-embedding state-of-the-art by up to 1.5 points absolute.

本文介绍了 SciRepEval，第一个全面评估科学文档表示方法的基准测试，可用于分类、回归、排名和搜索等多项任务，研究表明最新模型普遍存在跨任务格式泛化能力差的问题，但学习每个文档的多个嵌入以针对不同格式得到改善，实验发现多任务设置下的任务格式特定控制码和适配器优于现有的最新单嵌入模型。