In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs
for specific tasks by utilizing labeled examples as demonstrations in the
precondition prompts. Despite its promising performance, ICL suffers from
instability with the choice and arrangement of examples. Additionally, crafted
adversarial attacks pose a notable threat to the robustness of ICL. However,
existing attacks are either easy to detect, rely on external models, or lack
specificity towards ICL. To address these issues, this work introduces a novel
transferable attack for ICL, aiming to hijack LLMs to generate the targeted
response. The proposed LLM hijacking attack leverages a gradient-based prompt
search method to learn and append imperceptible adversarial suffixes to the
in-context demonstrations. Extensive experimental results on various tasks and
datasets demonstrate the effectiveness of our LLM hijacking attack, resulting
in a distracted attention towards adversarial tokens, consequently leading to
the targeted unwanted outputs.

通过引入一种新的对上下文学习的颠覆性攻击方法，本文展示了一种能够利用 LMLs 来生成针对性响应的方法，并通过对各种任务和数据集的广泛实验结果证明了其有效性。