The widespread adoption of Large Language Models (LLMs), exemplified by
OpenAI's ChatGPT, brings to the forefront the imperative to defend against
adversarial threats on these models. These attacks, which manipulate an LLM's
output by introducing malicious inputs, undermine the model's integrity and the
trust users place in its outputs. In response to this challenge, our paper
presents an innovative defensive strategy, given white box access to an LLM,
that harnesses residual activation analysis between transformer layers of the
LLM. We apply an established methodology for analyzing distinctive activation
patterns in the residual streams for a novel result of attack prompt
classification. We curate multiple datasets to demonstrate how this method of
classification has high accuracy across multiple types of attack scenarios,
including our newly-created attack dataset. Furthermore, we enhance the model's
resilience by integrating safety fine-tuning techniques for LLMs in order to
measure its effect on our capability to detect attacks. The results underscore
the effectiveness of our approach in enhancing the detection and mitigation of
adversarial inputs, advancing the security framework within which LLMs operate.

借助大型语言模型（LLMs），我们提出了一种创新的防御策略，通过对 LLM 的 Transformer 层之间的残余激活分析，实现对恶意输入的攻击提示分类的高准确性，同时集成安全微调技术提升模型的鲁棒性和提高检测和缓解对抗性输入的能力。