An important task for designing QA systems is answer sentence selection
(AS2): selecting the sentence containing (or constituting) the answer to a
question from a set of retrieved relevant documents. In this paper, we propose
three novel sentence-level transformer pre-training objectives that incorporate
paragraph-level semantics within and across documents, to improve the
performance of transformers for AS2, and mitigate the requirement of large
labeled datasets. Specifically, the model is tasked to predict whether: (i) two
sentences are extracted from the same paragraph, (ii) a given sentence is
extracted from a given paragraph, and (iii) two paragraphs are extracted from
the same document. Our experiments on three public and one industrial AS2
datasets demonstrate the empirical superiority of our pre-trained transformers
over baseline models such as RoBERTa and ELECTRA for AS2.

本文提出了三种新的句子级别的 transformer 预训练目标，以提高 transformer 模型在答案句子选择和文档检索任务中的表现，并减轻对大量标记数据的需求。实验证明，我们的预训练 transformer 模型在三个公共 AS2 数据集和一个工业 AS2 数据集上的表现优越，胜过 RoBERTa 和 ELECTRA 等基线模型。