Existing AI alignment approaches assume that preferences are static, which is
unrealistic: our preferences change, and may even be influenced by our
interactions with AI systems themselves. To clarify the consequences of
incorrectly assuming static preferences, we introduce Dynamic Reward Markov
Decision Processes (DR-MDPs), which explicitly model preference changes and the
AI's influence on them. We show that despite its convenience, the
static-preference assumption may undermine the soundness of existing alignment
techniques, leading them to implicitly reward AI systems for influencing user
preferences in ways users may not truly want. We then explore potential
solutions. First, we offer a unifying perspective on how an agent's
optimization horizon may partially help reduce undesirable AI influence. Then,
we formalize different notions of AI alignment that account for preference
change from the outset. Comparing the strengths and limitations of 8 such
notions of alignment, we find that they all either err towards causing
undesirable AI influence, or are overly risk-averse, suggesting that a
straightforward solution to the problems of changing preferences may not exist.
As there is no avoiding grappling with changing preferences in real-world
settings, this makes it all the more important to handle these issues with
care, balancing risks and capabilities. We hope our work can provide conceptual
clarity and constitute a first step towards AI alignment practices which
explicitly account for (and contend with) the changing and influenceable nature
of human preferences.

现有的 AI 对齐方法假设偏好是静态的，这是不现实的：我们的偏好会改变，甚至可能受到与 AI 系统的交互的影响。为了阐明错误地假设偏好是静态的后果，我们引入了动态回报马尔可夫决策过程 (DR-MDPs)，明确地模拟偏好变化和 AI 对其的影响。我们发现，尽管静态偏好的假设具有方便性，但它可能破坏现有对齐技术的准确性，使其暗地里奖励 AI 系统以影响用户偏好，而用户可能并不真正想要这样。然后，我们探讨了潜在的解决方案。首先，我们提供了一个统一的视角，阐述了一个代理的优化范围在某种程度上如何帮助减少不希望的 AI 影响。然后，我们从一开始就形式化了不同的 AI 对齐概念，考虑了偏好的变化。比较了 8 种这样的对齐概念的优缺点，发现它们要么倾向于引起不良的 AI 影响，要么过于风险回避，这表明解决偏好改变问题的简单解决方案可能不存在。由于在实际环境中无法避免处理不断变化的偏好，这使得我们更加重视如何平衡风险和能力来处理这些问题。我们希望我们的工作可以提供概念的清晰性，并成为针对人类偏好的变化性和可影响性明确考虑和应对的 AI 对齐实践的第一步。

与可变且可影响奖励函数保持 AI 对齐

AI Alignment with Changing and Influenceable Reward Functions

We demonstrate a proof-of-concept of a large language model conducting
corporate lobbying related activities. An autoregressive large language model
(OpenAI's text-davinci-003) determines if proposed U.S. Congressional bills are
relevant to specific public companies and provides explanations and confidence
levels. For the bills the model deems as relevant, the model drafts a letter to
the sponsor of the bill in an attempt to persuade the congressperson to make
changes to the proposed legislation. We use hundreds of novel ground-truth
labels of the relevance of a bill to a company to benchmark the performance of
the model. It outperforms the baseline of predicting the most common outcome of
irrelevance. We also benchmark the performance of the previous OpenAI GPT-3
model (text-davinci-002), which was the state-of-the-art model on many academic
natural language tasks until text-davinci-003 was recently released. The
performance of text-davinci-002 is worse than the simple baseline. Longer-term,
if AI begins to influence law in a manner that is not a direct extension of
human intentions, this threatens the critical role that law as information
could play in aligning AI with humans. Initially, AI is being used to simply
augment human lobbyists for a small portion of their daily tasks. However,
firms have an incentive to use less and less human oversight over automated
assessments of policy ideas and the written communication to regulatory
agencies and Congressional staffers. The core question raised is where to draw
the line between human-driven and AI-driven policy influence.

用大型语言模型辅助企业游说，帮助识别相关立法并草拟游说信，引发对人工智能游说的界限问题。