Finetuning Pretrained Language Models (PLM) for IR has been de facto the
standard practice since their breakthrough effectiveness few years ago. But, is
this approach well understood? In this paper, we study the impact of the
pretraining collection on the final IR effectiveness. In particular, we
challenge the current hypothesis that PLM shall be trained on a large enough
generic collection and we show that pretraining from scratch on the collection
of interest is surprisingly competitive with the current approach. We benchmark
first-stage ranking rankers and cross-encoders for reranking on the task of
general passage retrieval on MSMARCO, Mr-Tydi for Arabic, Japanese and Russian,
and TripClick for specific domain. Contrary to popular belief, we show that,
for finetuning first-stage rankers, models pretrained solely on their
collection have equivalent or better effectiveness compared to more general
models. However, there is a slight effectiveness drop for rerankers pretrained
only on the target collection. Overall, our study sheds a new light on the role
of the pretraining collection and should make our community ponder on building
specialized models by pretraining from scratch. Last but not least, doing so
could enable better control of efficiency, data bias and replicability, which
are key research questions for the IR community.

本文研究预训练语言模型（PLM）对信息检索有效性的影响，挑战了当前假设 PLM 应该在足够大的基础集合上进行训练的想法，并表明从感兴趣的集合开始预训练与当前方法相比具有令人惊讶的竞争力。作者对一些任务进行了性能基准测试，发现模型预训练只在目标集合上的效果略微下降。研究结论指出从头开始进行预训练能够更好地控制效率，数据偏差和可重复性，并促进了建立专业化模型的想法。