Counterfactual learning to rank (CLTR) has attracted extensive attention in
the IR community for its ability to leverage massive logged user interaction
data to train ranking models. While the CLTR models can be theoretically
unbiased when the user behavior assumption is correct and the propensity
estimation is accurate, their effectiveness is usually empirically evaluated
via simulation-based experiments due to a lack of widely-available,
large-scale, real click logs. However, the mainstream simulation-based
experiments are somewhat limited as they often feature a single, deterministic
production ranker and simplified user simulation models to generate the
synthetic click logs. As a result, the robustness of CLTR models in complex and
diverse situations is largely unknown and needs further investigation.
To address this problem, in this paper, we aim to investigate the robustness
of existing CLTR models in a reproducibility study with extensive
simulation-based experiments that (1) use both deterministic and stochastic
production rankers, each with different ranking performance, and (2) leverage
multiple user simulation models with different user behavior assumptions. We
find that the DLA models and IPS-DCM show better robustness under various
simulation settings than IPS-PBM and PRS with offline propensity estimation.
Besides, the existing CLTR models often fail to outperform the naive click
baselines when the production ranker has relatively high ranking performance or
certain randomness, which suggests an urgent need for developing new CLTR
algorithms that work for these settings.

通过广泛的基于仿真的实验，探究了现有的反事实学习排序模型在复杂和多样化情境下的稳健性，并发现 DLA 模型和 IPS-DCM 在各种仿真设置下展现出更好的稳健性，而 IPS-PBM 和 PRS 则较为脆弱；此外，当生产排序器具有较高的排序性能或存在一定的随机性时，现有的反事实学习排序模型往往无法超越简单的点击基准，因此迫切需要开发适用于这些情境的新的反事实学习排序算法。