An important aspect in developing language models that interact with humans
is aligning their behavior to be useful and unharmful for their human users.
This is usually achieved by tuning the model in a way that enhances desired
behaviors and inhibits undesired ones, a process referred to as alignment. In
this paper, we propose a theoretical approach called Behavior Expectation
Bounds (BEB) which allows us to formally investigate several inherent
characteristics and limitations of alignment in large language models.
Importantly, we prove that for any behavior that has a finite probability of
being exhibited by the model, there exist prompts that can trigger the model
into outputting this behavior, with probability that increases with the length
of the prompt. This implies that any alignment process that attenuates
undesired behavior but does not remove it altogether, is not safe against
adversarial prompting attacks. Furthermore, our framework hints at the
mechanism by which leading alignment approaches such as reinforcement learning
from human feedback increase the LLM's proneness to being prompted into the
undesired behaviors. Moreover, we include the notion of personas in our BEB
framework, and find that behaviors which are generally very unlikely to be
exhibited by the model can be brought to the front by prompting the model to
behave as specific persona. This theoretical result is being experimentally
demonstrated in large scale by the so called contemporary "chatGPT jailbreaks",
where adversarial users trick the LLM into breaking its alignment guardrails by
triggering it into acting as a malicious persona. Our results expose
fundamental limitations in alignment of LLMs and bring to the forefront the
need to devise reliable mechanisms for ensuring AI safety.

本文提出了一种名为行为期望范围（BEB）的理论方法，用于正式研究大型语言模型对齐的几种固有特性和限制，揭示了 LMM 对齐的根本局限性，并凸显了确保 AI 安全的可靠机制的必要性。