Large Language Models (LLMs) such as ChatGPT and Llama-2 have become
prevalent in real-world applications, exhibiting impressive text generation
performance. LLMs are fundamentally developed from a scenario where the input
data remains static and lacks a clear structure. To behave interactively over
time, LLM-based chat systems must integrate additional contextual information
(i.e., chat history) into their inputs, following a pre-defined structure. This
paper identifies how such integration can expose LLMs to misleading context
from untrusted sources and fail to differentiate between system and user
inputs, allowing users to inject context. We present a systematic methodology
for conducting context injection attacks aimed at eliciting disallowed
responses by introducing fabricated context. This could lead to illegal
actions, inappropriate content, or technology misuse. Our context fabrication
strategies, acceptance elicitation and word anonymization, effectively create
misleading contexts that can be structured with attacker-customized prompt
templates, achieving injection through malicious user messages. Comprehensive
evaluations on real-world LLMs such as ChatGPT and Llama-2 confirm the efficacy
of the proposed attack with success rates reaching 97%. We also discuss
potential countermeasures that can be adopted for attack detection and
developing more secure models. Our findings provide insights into the
challenges associated with the real-world deployment of LLMs for interactive
and structured data scenarios.

通过在聊天系统中引入虚构的上下文，利用大型语言模型中的错误分类和上下文混淆的问题，可以进行上下文注入攻击，破坏实时交互的大型语言模型的安全性。研究发现了进行上下文注入攻击的策略并验证了其高成功率，同时提出了攻击检测和开发更安全模型的可能对策。