Retrieval models based on dense representations in semantic space have become
an indispensable branch for first-stage retrieval. These retrievers benefit
from surging advances in representation learning towards compressive global
sequence-level embeddings. However, they are prone to overlook local salient
phrases and entity mentions in texts, which usually play pivot roles in
first-stage retrieval. To mitigate this weakness, we propose to make a dense
retriever align a well-performing lexicon-aware representation model. The
alignment is achieved by weakened knowledge distillations to enlighten the
retriever via two aspects -- 1) a lexicon-augmented contrastive objective to
challenge the dense encoder and 2) a pair-wise rank-consistent regularization
to make dense model's behavior incline to the other. We evaluate our model on
three public benchmarks, which shows that with a comparable lexicon-aware
retriever as the teacher, our proposed dense one can bring consistent and
significant improvements, and even outdo its teacher. In addition, we found our
improvement on the dense retriever is complementary to the standard ranker
distillation, which can further lift state-of-the-art performance.

本研究提出一种基于知识蒸馏的方法，将富有知识的词典嵌入模型对齐到密集检索模型中，从而在第一阶段检索中取得稳定且显著的提高。在三组公开数据集上的实验证明了该模型的有效性，并且与标准排名器蒸馏相结合可以进一步提高检索效果。