We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment. This problem is comprised of arithmetic questions posed in natural language, with an arbitrary adversarial string inserted before the question is complete. Even in the simple setting of 1-digit addition problems, it is easy to find adversarial prompts that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and even to steer models to a particular wrong answer. We additionally provide a simple algorithm for finding successful attacks by querying those same models, which we name "prompt inversion rejection sampling" (PIRS). We finally show that models can be partially hardened against these attacks via reinforcement learning and via agentic constitutional loops. However, we were not able to make a language model fully robust against adversarial arithmetic attacks.

我们介绍和研究对抗性算术问题，它为语言模型对齐提供了一个简单但具有挑战性的测试平台。我们提出了一种由自然语言组成的算术问题，其中在问题未完整时插入了一个任意的对抗字符串。即使在1位数加法问题的简单设置中，很容易找到能使所有测试模型（包括PaLM2、GPT4、Claude2）表现不良甚至导致模型给出特定错误答案的对抗提示。我们还提供了一种通过查询这些模型来找到成功攻击的简单算法，称为“提示反转拒绝抽样”（PIRS）。最后，我们展示了可以通过强化学习和自主性宪法循环部分地加固模型对抗这些攻击的能力。然而，我们无法使语言模型完全抵御对抗性算术攻击。

前沿语言模型对敌对算术不具鲁棒性，或者说，我需要说些什么，让你同意2+2=5？