Traditionally, sparse retrieval systems relied on lexical representations to
retrieve documents, such as BM25, dominated information retrieval tasks. With
the onset of pre-trained transformer models such as BERT, neural sparse
retrieval has led to a new paradigm within retrieval. Despite the success,
there has been limited software supporting different sparse retrievers running
in a unified, common environment. This hinders practitioners from fairly
comparing different sparse models and obtaining realistic evaluation results.
Another missing piece is, that a majority of prior work evaluates sparse
retrieval models on in-domain retrieval, i.e. on a single dataset: MS MARCO.
However, a key requirement in practical retrieval systems requires models that
can generalize well to unseen out-of-domain, i.e. zero-shot retrieval tasks. In
this work, we provide SPRINT, a unified Python toolkit based on Pyserini and
Lucene, supporting a common interface for evaluating neural sparse retrieval.
The toolkit currently includes five built-in models: uniCOIL, DeepImpact,
SPARTA, TILDEv2 and SPLADEv2. Users can also easily add customized models by
defining their term weighting method. Using our toolkit, we establish strong
and reproducible zero-shot sparse retrieval baselines across the
well-acknowledged benchmark, BEIR. Our results demonstrate that SPLADEv2
achieves the best average score of 0.470 nDCG@10 on BEIR amongst all neural
sparse retrievers. In this work, we further uncover the reasons behind its
performance gain. We show that SPLADEv2 produces sparse representations with a
majority of tokens outside of the original query and document which is often
crucial for its performance gains, i.e. a limitation among its other sparse
counterparts. We provide our SPRINT toolkit, models, and data used in our
experiments publicly here at this https URL

SPRINT 是一个基于 Pyserini 和 Lucene 的统一 Python 工具包，支持评估神经稀疏检索。通过在 BEIR 上建立强大且可复现的零样本稀疏检索基线，我们的结果表明 SPLADEv2 在所有神经稀疏检索器中取得了最佳的平均 0.470 nDCG@10 分数。