Approaches to recommendation are typically evaluated in one of two ways: (1)
via a (simulated) online experiment, often seen as the gold standard, or (2)
via some offline evaluation procedure, where the goal is to approximate the
outcome of an online experiment. Several offline evaluation metrics have been
adopted in the literature, inspired by ranking metrics prevalent in the field
of Information Retrieval. (Normalised) Discounted Cumulative Gain (nDCG) is one
such metric that has seen widespread adoption in empirical studies, and higher
(n)DCG values have been used to present new methods as the state-of-the-art in
top-$n$ recommendation for many years.
Our work takes a critical look at this approach, and investigates when we can
expect such metrics to approximate the gold standard outcome of an online
experiment. We formally present the assumptions that are necessary to consider
DCG an unbiased estimator of online reward and provide a derivation for this
metric from first principles, highlighting where we deviate from its
traditional uses in IR. Importantly, we show that normalising the metric
renders it inconsistent, in that even when DCG is unbiased, ranking competing
methods by their normalised DCG can invert their relative order. Through a
correlation analysis between off- and on-line experiments conducted on a
large-scale recommendation platform, we show that our unbiased DCG estimates
strongly correlate with online reward, even when some of the metric's inherent
assumptions are violated. This statement no longer holds for its normalised
variant, suggesting that nDCG's practical utility may be limited.

对推荐方法的评估方法进行了探讨，研究了离线评估和在线试验之间的关系，并分析了离线评估指标的一致性和实用性。

关于（标准化的）折扣累积收益作为一个离线评估 $n$ 个项目推荐的度量

On (Normalised) Discounted Cumulative Gain as an Offline Evaluation  Metric for Top-$n$ Recommendation

Algorithmic decision-making systems are increasingly used throughout the
public and private sectors to make important decisions or assist humans in
making these decisions with real social consequences. While there has been
substantial research in recent years to build fair decision-making algorithms,
there has been less research seeking to understand the factors that affect
people's perceptions of fairness in these systems, which we argue is also
important for their broader acceptance. In this research, we conduct an online
experiment to better understand perceptions of fairness, focusing on three sets
of factors: algorithm outcomes, algorithm development and deployment
procedures, and individual differences. We find that people rate the algorithm
as more fair when the algorithm predicts in their favor, even surpassing the
negative effects of describing algorithms that are very biased against
particular demographic groups. We find that this effect is moderated by several
variables, including participants' education level, gender, and several aspects
of the development procedure. Our findings suggest that systems that evaluate
algorithmic fairness through users' feedback must consider the possibility of
outcome favorability bias.

本研究使用在线实验的方法，探讨三组因素（算法预测结果、算法开发和部署程序以及个体差异）对算法公平性认知的影响，研究发现，参与者倾向于认为算法在其预测结果有利的情况下更加公平，这种影响受到多种变量的调节，如教育水平、性别以及开发程序的若干方面，研究结果表明在通过用户反馈评估算法公平性时，必须考虑预测结果偏好的可能存在。

算法决策中影响公平感知的因素：算法结果、开发过程和个体差异

Factors Influencing Perceived Fairness in Algorithmic Decision-Making:  Algorithm Outcomes, Development Procedures, and Individual Differences

Bots are, for many Web and social media users, the source of many dangerous
attacks or the carrier of unwanted messages, such as spam. Nevertheless,
crawlers and software agents are a precious tool for analysts, and they are
continuously executed to collect data or to test distributed applications.
However, no one knows which is the real potential of a bot whose purpose is to
control a community, to manipulate consensus, or to influence user behavior. It
is commonly believed that the better an agent simulates human behavior in a
social network, the more it can succeed to generate an impact in that
community. We contribute to shed light on this issue through an online social
experiment aimed to study to what extent a bot with no trust, no profile, and
no aims to reproduce human behavior, can become popular and influential in a
social media. Results show that a basic social probing activity can be used to
acquire social relevance on the network and that the so-acquired popularity can
be effectively leveraged to drive users in their social connectivity choices.
We also register that our bot activity unveiled hidden social polarization
patterns in the community and triggered an emotional response of individuals
that brings to light subtle privacy hazards perceived by the user base.

通过在线社交实验，发现没有信任、档案和复制用户行为目标的机器人可通过基本的社交探测活动在社交媒体上获得社交相关性，进而有效地引导用户的社交连接选择，并揭示了隐蔽的社交极化模式和用户感知的微妙隐私风险。