Is it possible for machines to think like humans? And if it is, how should we
go about teaching them to do so? As early as 1950, Alan Turing stated that we
ought to teach machines in the way of teaching a child. Reinforcement learning
with human feedback (RLHF) has emerged as a strong candidate toward allowing
agents to learn from human feedback in a naturalistic manner. RLHF is distinct
from traditional reinforcement learning as it provides feedback from a human
teacher in addition to a reward signal. It has been catapulted into public view
by multiple high-profile AI applications, including OpenAI's ChatGPT,
DeepMind's Sparrow, and Anthropic's Claude. These highly capable chatbots are
already overturning our understanding of how AI interacts with humanity. The
wide applicability and burgeoning success of RLHF strongly motivate the need to
evaluate its social impacts. In light of recent developments, this paper
considers an important question: can RLHF be developed and used without
negatively affecting human societies? Our objectives are threefold: to provide
a systematic study of the social effects of RLHF; to identify key social and
ethical issues of RLHF; and to discuss social impacts for stakeholders.
Although text-based applications of RLHF have received much attention, it is
crucial to consider when evaluating its social implications the diverse range
of areas to which it may be deployed. We describe seven primary ways in which
RLHF-based technologies will affect society by positively transforming human
experiences with AI. This paper ultimately proposes that RLHF has potential to
net positively impact areas of misinformation, AI value-alignment, bias, AI
access, cross-cultural dialogue, industry, and workforce. As RLHF raises
concerns that echo those of existing AI technologies, it will be important for
all to be aware and intentional in the adoption of RLHF.

本文系统地研究了 RLHF 的社会影响，提出了七种 RLHF 的应用方式，分析了其对人类体验 AI 的积极影响，同时也探讨了 RLHF 可能带来的社会道德问题，以及其与人类相互作用的方面。因为 RLHF 引发的关注类似于现有 AI 技术引发的关注，因此需要对 RLHF 的采用进行审慎且有意识的考虑。