Unbiased learning-to-rank (ULTR) is a well-established framework for learning
from user clicks, which are often biased by the ranker collecting the data.
While theoretically justified and extensively tested in simulation, ULTR
techniques lack empirical validation, especially on modern search engines. The
dataset released for the WSDM Cup 2023, collected from Baidu's search engine,
offers a rare opportunity to assess the real-world performance of prominent
ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the
subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed
improvements stem from applying ULTR or other learning techniques. We revisit
and extend the available experiments. We find that unbiased learning-to-rank
techniques do not bring clear performance improvements, especially compared to
the stark differences brought by the choice of ranking loss and query-document
features. Our experiments reveal that ULTR robustly improves click prediction.
However, these gains in click prediction do not translate to enhanced ranking
performance on expert relevance annotations, implying that conclusions strongly
depend on how success is measured in this benchmark.

通过对百度搜索引擎的实时数据集开展实证研究，发现无偏学习排序技术在点击预测方面有明显的性能提升，但这并不转化为在专家相关性标注中的排名性能改进，表明结论在很大程度上取决于基准评估方法。

无偏学习排名遇到现实：来自百度大规模搜索数据集的教训

Unbiased Learning to Rank Meets Reality: Lessons from Baidu's  Large-Scale Search Dataset

In modern recommendation systems, unbiased learning-to-rank (LTR) is crucial
for prioritizing items from biased implicit user feedback, such as click data.
Several techniques, such as Inverse Propensity Weighting (IPW), have been
proposed for single-sided markets. However, less attention has been paid to
two-sided markets, such as job platforms or dating services, where successful
conversions require matching preferences from both users. This paper addresses
the complex interaction of biases between users in two-sided markets and
proposes a tailored LTR approach. We first present a formulation of feedback
mechanisms in two-sided matching platforms and point out that their implicit
feedback may include position bias from both user groups. On the basis of this
observation, we extend the IPW estimator and propose a new estimator, named
two-sided IPW, to address the position bases in two-sided markets. We prove
that the proposed estimator satisfies the unbiasedness for the ground-truth
ranking metric. We conducted numerical experiments on real-world two-sided
platforms and demonstrated the effectiveness of our proposed method in terms of
both precision and robustness. Our experiments showed that our method
outperformed baselines especially when handling rare items, which are less
frequently observed in the training data.

本文针对两边市场中用户间的偏见相互作用问题，提出了一个定制的无偏学习排序方法，证明了该方法满足真实排名度量的无偏性，并通过实验证明了该方法在处理稀有物品时优于基线方法。