Recent generative AI systems have demonstrated more advanced persuasive
capabilities and are increasingly permeating areas of life where they can
influence decision-making. Generative AI presents a new risk profile of
persuasion due the opportunity for reciprocal exchange and prolonged
interactions. This has led to growing concerns about harms from AI persuasion
and how they can be mitigated, highlighting the need for a systematic study of
AI persuasion. The current definitions of AI persuasion are unclear and related
harms are insufficiently studied. Existing harm mitigation approaches
prioritise harms from the outcome of persuasion over harms from the process of
persuasion. In this paper, we lay the groundwork for the systematic study of AI
persuasion. We first put forward definitions of persuasive generative AI. We
distinguish between rationally persuasive generative AI, which relies on
providing relevant facts, sound reasoning, or other forms of trustworthy
evidence, and manipulative generative AI, which relies on taking advantage of
cognitive biases and heuristics or misrepresenting information. We also put
forward a map of harms from AI persuasion, including definitions and examples
of economic, physical, environmental, psychological, sociocultural, political,
privacy, and autonomy harm. We then introduce a map of mechanisms that
contribute to harmful persuasion. Lastly, we provide an overview of approaches
that can be used to mitigate against process harms of persuasion, including
prompt engineering for manipulation classification and red teaming. Future work
will operationalise these mitigations and study the interaction between
different types of mechanisms of persuasion.

最近的生成型 AI 系统展示了更先进的说服能力，并且越来越多地渗透到可以影响决策的领域。生成型 AI 带来了新的说服风险概况，因为它提供了相互交流和长时间互动的机会。这引起了人们对 AI 说服的伤害的忧虑，以及如何减轻这些伤害的需求，凸显出了对 AI 说服进行系统研究的需求。本文为 AI 说服的系统研究奠定了基础。我们首先提出了有关生成型 AI 的定义。我们区分了基于提供相关事实、合理推理或其他形式的值得信赖的证据的理性说服型生成型 AI，以及基于利用认知偏差和启发法或者歪曲信息的操纵型生成型 AI。我们还提出了 AI 说服伤害的一张风险图，包括经济的、身体的、环境的、心理的、社会文化的、政治的、隐私的和自主权的伤害的定义和示例。然后，我们介绍了有助于有害说服的机制图。最后，我们提供了一种用于减轻说服过程伤害的方法概述，包括用于操纵分类和红队测试的提示工程。未来工作将使这些减轻措施具体化，并研究不同类型的说服机制之间的相互作用。