Reinforcement learning from human feedback (RLHF) has been an effective
technique for aligning AI systems with human values, with remarkable successes
in fine-tuning large-language models recently. Most existing RLHF paradigms
make the underlying assumption that human preferences are relatively
homogeneous, and can be encoded by a single reward model. In this paper, we
focus on addressing the issues due to the inherent heterogeneity in human
preferences, as well as their potential strategic behavior in providing
feedback. Specifically, we propose two frameworks to address heterogeneous
human feedback in principled ways: personalization-based one and
aggregation-based one. For the former, we propose two approaches based on
representation learning and clustering, respectively, for learning multiple
reward models that trades off the bias (due to preference heterogeneity) and
variance (due to the use of fewer data for learning each model by
personalization). We then establish sample complexity guarantees for both
approaches. For the latter, we aim to adhere to the single-model framework, as
already deployed in the current RLHF paradigm, by carefully aggregating diverse
and truthful preferences from humans. We propose two approaches based on reward
and preference aggregation, respectively: the former utilizes both
utilitarianism and Leximin approaches to aggregate individual reward models,
with sample complexity guarantees; the latter directly aggregates the human
feedback in the form of probabilistic opinions. Under the
probabilistic-opinion-feedback model, we also develop an approach to handle
strategic human labelers who may bias and manipulate the aggregated preferences
with untruthful feedback. Based on the ideas in mechanism design, our approach
ensures truthful preference reporting, with the induced aggregation rule
maximizing social welfare functions.

利用个性化和聚合两个框架解决存在异质人类反馈的增强学习中的问题并确保较高的样本效率。

通过个性化和偏好聚合来自异构反馈的基于原则的 RLHF

Principled RLHF from Heterogeneous Feedback via Personalization and  Preference Aggregation

The way that people make choices or exhibit preferences can be strongly
affected by the set of available alternatives, often called the choice set.
Furthermore, there are usually heterogeneous preferences, either at an
individual level within small groups or within sub-populations of large groups.
Given the availability of choice data, there are now many models that capture
this behavior in order to make effective predictions--however, there is little
work in understanding how directly changing the choice set can be used to
influence the preferences of a collection of decision-makers. Here, we use
discrete choice modeling to develop an optimization framework of such
interventions for several problems of group influence, namely maximizing
agreement or disagreement and promoting a particular choice. We show that these
problems are NP-hard in general, but imposing restrictions reveals a
fundamental boundary: promoting a choice can be easier than encouraging
consensus or sowing discord. We design approximation algorithms for the hard
problems and show that they work well on real-world choice data.

本文基于离散选择建模，针对整体权衡、最大化一致性或不一致性以及促进特定选择等问题，提出了一种优化框架，着重研究了直接更改选择集合对决策者群体偏好的影响，通过限制条件的引入，揭示了不同问题之间的基本边界，为难以解决的问题设计了近似算法，并展示了这些算法在现实世界的应用。

基于群体决策离散选择模型的选择集优化

Choice Set Optimization Under Discrete Choice Models of Group Decisions

Ranking algorithms play a crucial role in online platforms ranging from
search engines to recommender systems. In this paper, we identify a surprising
consequence of popularity-based rankings: the fewer the items reporting a given
signal, the higher the share of the overall traffic they collectively attract.
This few-get-richer effect emerges in settings where there are few distinct
classes of items (e.g., left-leaning news sources versus right-leaning news
sources), and items are ranked based on their popularity. We demonstrate
analytically that the few-get-richer effect emerges when people tend to click
on top-ranked items and have heterogeneous preferences for the classes of
items. Using simulations, we analyze how the strength of the effect changes
with assumptions about the setting and human behavior. We also test our
predictions experimentally in an online experiment with human participants. Our
findings have important implications to understand the spread of
misinformation.

本文研究了基于流行度的排序算法所导致的少数热门效应以及其在信息传播方面的重要影响。通过理论和实验分析，发现人们点击排名靠前的物品，并具有不同的物品偏好，导致物品流量不平衡，进而影响了信息的传播。

基于人气排名的寡头富者现象

The few-get-richer: a surprising consequence of popularity-based  rankings

Many recommendation algorithms rely on user data to generate recommendations.
However, these recommendations also affect the data obtained from future users.
This work aims to understand the effects of this dynamic interaction. We
propose a simple model where users with heterogeneous preferences arrive over
time. Based on this model, we prove that naive estimators, i.e. those which
ignore this feedback loop, are not consistent. We show that consistent
estimators are efficient in the presence of myopic agents. Our results are
validated using extensive simulations.

该研究旨在了解推荐算法对未来用户数据的影响，通过提出一个简单模型，表明忽略这个反馈循环将导致非一致性评估器，因此验证了我们的结论。