Learning of preference models from human feedback has been central to recent
advances in artificial intelligence. Motivated by this progress, and the cost
of obtaining high-quality human annotations, we study the problem of data
collection for learning preference models. The key idea in our work is to
generalize optimal designs, a tool for computing efficient data logging
policies, to ranked lists. To show the generality of our ideas, we study both
absolute and relative feedback on items in the list. We design efficient
algorithms for both settings and analyze them. We prove that our preference
model estimators improve with more data and so does the ranking error under the
estimators. Finally, we experiment with several synthetic and real-world
datasets to show the statistical efficiency of our algorithms.

从人类反馈中学习偏好模型一直是人工智能领域最近进展的核心。本研究通过推广最优设计的概念，研究了用于学习偏好模型的数据收集问题，并提出了面向排名列表的有效算法，证明了模型估计器随更多数据而改善，估计器下的排名误差也随之减少，并在多个合成和真实数据集上进行实验以展示算法的统计效率。

人类反馈的最佳设计

Optimal Design for Human Feedback

In this work, we introduce a novel metric for auditing group fairness in
ranked lists. Our approach offers two benefits compared to the state of the
art. First, we offer a blueprint for modeling of user attention. Rather than
assuming a logarithmic loss in importance as a function of the rank, we can
account for varying user behaviors through parametrization. For example, we
expect a user to see more items during a viewing of a social media feed than
when they inspect the results list of a single web search query. Second, we
allow non-binary protected attributes to enable investigating inherently
continuous attributes (\eg political alignment on the liberal to conservative
spectrum) as well as to facilitate measurements across aggregated sets of
search results, rather than separately for each result list. By combining these
two elements into our metric, we are able to better address the human factors
inherent in this problem. We measure the whole sociotechnical system,
consisting of a ranking algorithm and individuals using it, instead of
exclusively focusing on the ranking algorithm. Finally, we use our metric to
perform three simulated fairness audits. We show that determining fairness of a
ranked output necessitates knowledge (or a model) of the end-users of the
particular service. Depending on their attention distribution function, a fixed
ranking of results can appear biased both in favor and against a protected
group.

本文提出了一种用于审计排名列表中群体公平性的新度量，通过我们的度量方法，能够更好地解决人为因素对公平性评估的影响，同时考虑了用户的注意力模式以及对保护属性的支持，我们还利用度量方法对三个模拟案例进行了公平性审计。

用户注意力对排名列表中公平群组表现的影响量化

Quantifying the Impact of User Attention on Fair Group Representation in  Ranked Lists

Many web systems rank and present a list of items to users, from recommender
systems to search and advertising. An important problem in practice is to
evaluate new ranking policies offline and optimize them before they are
deployed. We address this problem by proposing evaluation algorithms for
estimating the expected number of clicks on ranked lists from historical logged
data. The existing algorithms are not guaranteed to be statistically efficient
in our problem because the number of recommended lists can grow exponentially
with their length. To overcome this challenge, we use models of user
interaction with the list of items, the so-called click models, to construct
estimators that learn statistically efficiently. We analyze our estimators and
prove that they are more efficient than the estimators that do not use the
structure of the click model, under the assumption that the click model holds.
We evaluate our estimators in a series of experiments on a real-world dataset
and show that they consistently outperform prior estimators.

本文提出了一种评估算法来预测历史日志数据中排名列表上的点击数，并使用用户与项目列表的交互模型来构建统计效率更高的估计器。实验结果表明，相对于先前的估计器，该算法具有更高的性能表现。