Pretrained large language models (LLMs) are becoming increasingly powerful
and ubiquitous in mainstream applications such as being a personal assistant, a
dialogue model, etc. As these models become proficient in deducing user
preferences and offering tailored assistance, there is an increasing concern
about the ability of these models to influence, modify and in the extreme case
manipulate user preference adversarially. The issue of lack of interpretability
in these models in adversarial settings remains largely unsolved. This work
tries to study adversarial behavior in user preferences from the lens of
attention probing, red teaming and white-box analysis. Specifically, it
provides a bird's eye view of existing literature, offers red teaming samples
for dialogue models like ChatGPT and GODEL and probes the attention mechanism
in the latter for non-adversarial and adversarial settings.

本研究从注意力探测、红队作战和白盒分析等多个角度，探讨预训练大型语言模型在用户喜好中的敌对行为，并针对 ChatGPT 和 GODEL 等对话模型提供红队样本，同时探究后者在非对抗性和对抗性环境下的注意力机制。