The unbiased learning to rank (ULTR) problem has been greatly advanced by
recent deep learning techniques and well-designed debias algorithms. However,
promising results on the existing benchmark datasets may not be extended to the
practical scenario due to the following disadvantages observed from those
popular benchmark datasets: (1) outdated semantic feature extraction where
state-of-the-art large scale pre-trained language models like BERT cannot be
exploited due to the missing of the original text;(2) incomplete display
features for in-depth study of ULTR, e.g., missing the displayed abstract of
documents for analyzing the click necessary bias; (3) lacking real-world user
feedback, leading to the prevalence of synthetic datasets in the empirical
study. To overcome the above disadvantages, we introduce the Baidu-ULTR
dataset. It involves randomly sampled 1.2 billion searching sessions and 7,008
expert annotated queries, which is orders of magnitude larger than the existing
ones. Baidu-ULTR provides:(1) the original semantic feature and a pre-trained
language model for easy usage; (2) sufficient display information such as
position, displayed height, and displayed abstract, enabling the comprehensive
study of different biases with advanced techniques such as causal discovery and
meta-learning; and (3) rich user feedback on search result pages (SERPs) like
dwelling time, allowing for user engagement optimization and promoting the
exploration of multi-task learning in ULTR. In this paper, we present the
design principle of Baidu-ULTR and the performance of benchmark ULTR algorithms
on this new data resource, favoring the exploration of ranking for long-tail
queries and pre-training tasks for ranking. The Baidu-ULTR dataset and
corresponding baseline implementation are available at
this https URL.

介绍了一个新的数据集 Baidu-ULTR 和基准 ULTR 算法在其上的性能，有利于长尾查询的排名和排名的预训练任务。