The dual-encoder has become the de facto architecture for dense retrieval.
Typically, it computes the latent representations of the query and document
independently, thus failing to fully capture the interactions between the query
and document. To alleviate this, recent work expects to get query-informed
representations of documents. During training, it expands the document with a
real query, while replacing the real query with a generated pseudo query at
inference. This discrepancy between training and inference makes the dense
retrieval model pay more attention to the query information but ignore the
document when computing the document representation. As a result, it even
performs worse than the vanilla dense retrieval model, since its performance
depends heavily on the relevance between the generated queries and the real
query. In this paper, we propose a curriculum sampling strategy, which also
resorts to the pseudo query at training and gradually increases the relevance
of the generated query to the real query. In this way, the retrieval model can
learn to extend its attention from the document only to both the document and
query, hence getting high-quality query-informed document representations.
Experimental results on several passage retrieval datasets show that our
approach outperforms the previous dense retrieval methods1.

本文提出了一种课程抽样策略来对原始的双编码器的不足进行改进，使得检索模型能够学习将注意力从文档扩展至文档和查询，从而获得高质量的查询知情文档表示，实验结果表明，我们的方法优于以前的密集检索方法。