Term-based sparse representations dominate the first-stage text retrieval in
industrial applications, due to its advantage in efficiency, interpretability,
and exact term matching. In this paper, we study the problem of transferring
the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse
representations, aiming to improve the representation capacity of
bag-of-words(BoW) method for semantic-level matching, while still keeping its
advantages. Specifically, we propose a novel framework SparTerm to directly
learn sparse text representations in the full vocabulary space. The proposed
SparTerm comprises an importance predictor to predict the importance for each
term in the vocabulary, and a gating controller to control the term activation.
These two modules cooperatively ensure the sparsity and flexibility of the
final text representation, which unifies the term-weighting and expansion in
the same framework. Evaluated on MSMARCO dataset, SparTerm significantly
outperforms traditional sparse methods and achieves state of the art ranking
performance among all the PLM-based sparse models.

本文提出了一种名为 SparTerm 的框架，旨在将预训练语言模型的深度知识转移至基于词袋法的稀疏表示，以提高其语义级匹配的表达能力，同时保持其效率、可解释性和准确性，并在 MSMARCO 数据集上获得了 state of the art 的排名性能。