Large language model systems face important security risks from maliciously
crafted messages that aim to overwrite the system's original instructions or
leak private data. To study this problem, we organized a capture-the-flag
competition at IEEE SaTML 2024, where the flag is a secret string in the LLM
system prompt. The competition was organized in two phases. In the first phase,
teams developed defenses to prevent the model from leaking the secret. During
the second phase, teams were challenged to extract the secrets hidden for
defenses proposed by the other teams. This report summarizes the main insights
from the competition. Notably, we found that all defenses were bypassed at
least once, highlighting the difficulty of designing a successful defense and
the necessity for additional research to protect LLM systems. To foster future
research in this direction, we compiled a dataset with over 137k multi-turn
attack chats and open-sourced the platform.

大型语言模型系统在安全方面面临重要风险，如恶意设计的消息试图覆盖系统的原始指令或泄露私人数据。为了研究这个问题，我们在 IEEE SaTML 2024 组织了一场夺旗赛，其中旗标是 LLM 系统提示中的机密字符串。该比赛分为两个阶段，第一阶段是团队开发防御措施以防止模型泄露机密信息，第二阶段是团队挑战提取其他团队提出的防御措施中隐藏的机密信息。本报告总结了比赛的主要见解，发现所有防御措施至少被绕过一次，凸显了设计成功防御措施的难度以及进一步研究保护 LLM 系统的必要性。为了促进未来在这个方向的研究，我们编制了一个包含超过 137k 个多轮次攻击对话的数据集，并开放源代码平台。