Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public
leaderboard such as MS MARCO, are intended to encourage research and track our
progress, addressing big questions in our field. However, the goal is not
simply to identify which run is "best", achieving the top score. The goal is to
move the field forward by developing new robust techniques, that work in many
different settings, and are adopted in research and practice. This paper uses
the MS MARCO and TREC Deep Learning Track as our case study, comparing it to
the case of TREC ad hoc ranking in the 1990s. We show how the design of the
evaluation effort can encourage or discourage certain outcomes, and raising
questions about internal and external validity of results. We provide some
analysis of certain pitfalls, and a statement of best practices for avoiding
such pitfalls. We summarize the progress of the effort so far, and describe our
desired end state of "robust usefulness", along with steps that might be
required to get us there.

通过比较 MS MARCO 和 TREC 深度学习赛道与 1990 年代的 TREC 专家检索的情况，本文探讨了评估努力的设计如何鼓励或阻碍某些结果，提出避免这些困难的最佳实践，概述了该领域的研究进展，并描述了我们对于 “健壮有用性” 的期望结束状态。

大数据环境下排名模型基准测试

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

Two step document ranking, where the initial retrieval is done by a classical
information retrieval method, followed by neural re-ranking model, is the new
standard. The best performance is achieved by using transformer-based models as
re-rankers, e.g., BERT. We employ Longformer, a BERT-like model for long
documents, on the MS MARCO document re-ranking task. The complete code used for
training the model can be found on:
this https URL

本文研究了使用传统信息检索方法进行初检索，然后采用神经网络再排序模型的两步文档排序，使用了适合长文档的 BERT 模型变种 Longformer 在 MS MARCO 文档再排序任务上获得了最佳性能。

使用 Longformer 进行 MS MARCO 文档重排任务

Longformer for MS MARCO Document Re-ranking Task

This paper studies the performances and behaviors of BERT in ranking tasks.
We explore several different ways to leverage the pre-trained BERT and
fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web
Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the
strong effectiveness of BERT in question-answering focused passage ranking
tasks, as well as the fact that BERT is a strong interaction-based seq2seq
matching model. Experimental results on TREC show the gaps between the BERT
pre-trained on surrounding contexts and the needs of ad hoc document ranking.
Analyses illustrate how BERT allocates its attentions between query-document
tokens in its Transformer layers, how it prefers semantic matches between
paraphrase tokens, and how that differs with the soft match patterns learned by
a click-trained neural ranker.

本文研究了预先训练的 BERT 在排名任务中的表现和行为，探索了在两种排名任务中优化 BERT 的几种不同方法，并实验证明了 BERT 在问题回答焦点的段落排名任务中的强有效性，以及 BERT 作为强交互式序列到序列匹配模型的事实。