Local explanations of learning-to-rank (LTR) models are thought to extract
the most important features that contribute to the ranking predicted by the LTR
model for a single data point. Evaluating the accuracy of such explanations is
challenging since the ground truth feature importance scores are not available
for most modern LTR models. In this work, we propose a systematic evaluation
technique for explanations of LTR models. Instead of using black-box models,
such as neural networks, we propose to focus on tree-based LTR models, from
which we can extract the ground truth feature importance scores using decision
paths. Once extracted, we can directly compare the ground truth feature
importance scores to the feature importance scores generated with explanation
techniques. We compare two recently proposed explanation techniques for LTR
models when using decision trees and gradient boosting models on the MQ2008
dataset. We show that the explanation accuracy in these techniques can largely
vary depending on the explained model and even which data point is explained.

本篇论文旨在实现一种系统评估 LTR 模型解释技术的方法，即通过决策树焕发的 ground truth 特征重要性得分，和这些解释技术产生的特征重要性得分进行直接比较，以此来比较 MQ2008 数据集上的 LTR 模型分别采用决策树和梯度提升模型时这两种解释技术的准确性，并表明这些技术的解释准确性很大程度上取决于所解释的模型和解释的数据点。

使用决策路径评估学习排名模型的本地无模型相关解释

Evaluating Local Model-Agnostic Explanations of Learning to Rank Models  with Decision Paths

The limited availability of ground truth relevance labels has been a major
impediment to the application of supervised methods to ad-hoc retrieval. As a
result, unsupervised scoring methods, such as BM25, remain strong competitors
to deep learning techniques which have brought on dramatic improvements in
other domains, such as computer vision and natural language processing. Recent
works have shown that it is possible to take advantage of the performance of
these unsupervised methods to generate training data for learning-to-rank
models. The key limitation to this line of work is the size of the training set
required to surpass the performance of the original unsupervised method, which
can be as large as $10^{13}$ training examples. Building on these insights, we
propose two methods to reduce the amount of training data required. The first
method takes inspiration from crowdsourcing, and leverages multiple
unsupervised rankers to generate soft, or noise-aware, training labels. The
second identifies harmful, or mislabeled, training examples and removes them
from the training set. We show that our methods allow us to surpass the
performance of the unsupervised baseline with far fewer training examples than
previous works.

本文介绍了两种减少训练数据需求的方法：一种是通过众包方法生成标签，并去除有害数据；另一种则是使用学习排序模型生成训练数据。这些方法使我们要比以前更少的训练数据即可超越无监督基准的表现。

从少到多学更多：加强弱监督以实现即时检索

Learning More From Less: Towards Strengthening Weak Supervision for  Ad-Hoc Retrieval

Predictive models are omnipresent in automated and assisted decision making
scenarios. But for the most part they are used as black boxes which output a
prediction without understanding partially or even completely how different
features influence the model prediction avoiding algorithmic transparency.
Rankings are ordering over items encoding implicit comparisons typically
learned using a family of features using learning-to-rank models. In this paper
we focus on how best we can understand the decisions made by a ranker in a
post-hoc model agnostic manner. We operate on the notion of interpretability
based on explainability of rankings over an interpretable feature space.
Furthermore we train a tree based model (inherently interpretable) using labels
from the ranker, called secondary training data to provide explanations.
Consequently, we attempt to study how well does a subset of features,
potentially interpretable, explain the full model under different training
sizes and algorithms. We do experiments on the learning to rank datasets with
30k queries and report results that serve show in certain settings we can learn
a faithful interpretable ranker.

本文研究了如何在后期操作中用可解释的特征空间解释排名模型，并使用来自排名器的标签来训练树状模型，以提供解释，研究表明在某些情况下，可以学习到忠实且可解释的排名器。