Prompts play a crucial role in guiding the responses of Large Language Models
(LLMs). However, the intricate role of individual tokens in prompts, known as
input saliency, in shaping the responses remains largely underexplored.
Existing saliency methods either misalign with LLM generation objectives or
rely heavily on linearity assumptions, leading to potential inaccuracies. To
address this, we propose Token Distribution Dynamics (TDD), a
\textcolor{black}{simple yet effective} approach to unveil and manipulate the
role of prompts in generating LLM outputs. TDD leverages the robust
interpreting capabilities of the language model head (LM head) to assess input
saliency. It projects input tokens into the embedding space and then estimates
their significance based on distribution dynamics over the vocabulary. We
introduce three TDD variants: forward, backward, and bidirectional, each
offering unique insights into token relevance. Extensive experiments reveal
that the TDD surpasses state-of-the-art baselines with a big margin in
elucidating the causal relationships between prompts and LLM outputs. Beyond
mere interpretation, we apply TDD to two prompt manipulation tasks for
controlled text generation: zero-shot toxic language suppression and sentiment
steering. Empirical results underscore TDD's proficiency in identifying both
toxic and sentimental cues in prompts, subsequently mitigating toxicity or
modulating sentiment in the generated content.

通过使用 Token Distribution Dynamics 方法，我们揭示了 Prompt 输入对于生成大型语言模型 (如 GPT) 输出的重要作用，该方法显著优于现有的基线方法，并且在控制文本生成方面具有潜力应用于抑制有害语言和调节情感。

揭示和操控大型语言模型中的提示影响

Unveiling and Manipulating Prompt Influence in Large Language Models

Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter inappropriate
images/prompts or remove undesirable concepts/styles via model fine-tuning, the
reliability of these safety mechanisms against diversified problematic prompts
remains largely unexplored. In this work, we propose Prompting4Debugging (P4D)
as a debugging and red-teaming tool that automatically finds problematic
prompts for diffusion models to test the reliability of a deployed safety
mechanism. We demonstrate the efficacy of our P4D tool in uncovering new
vulnerabilities of SD models with safety mechanisms. Particularly, our result
shows that around half of prompts in existing safe prompting benchmarks which
were originally considered "safe" can actually be manipulated to bypass many
deployed safety mechanisms, including concept removal, negative prompt, and
safety guidance. Our findings suggest that, without comprehensive testing, the
evaluations on limited safe prompting benchmarks can lead to a false sense of
safety for text-to-image models.

利用 Prompting4Debugging (P4D) 工具，我们发现了 Stable Diffusion（SD）模型中的新脆弱性，显示出许多原先被认为 “安全” 的提示实际上可以绕过许多已部署的安全机制，包括概念删除、负面提示和安全指导。这些发现表明，在没有全面测试的情况下，对有限的安全提示基准进行评估可能导致对文本到图像模型的安全性产生错误的认识。

Prompting4Debugging: 通过找到问题提示来进行 Red-Teaming 文本到图像传播模型 (Debugging4Prompting: Red-Teaming Text-to-Image Diffusion Models)

Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by  Finding Problematic Prompts

Recent explorations with commercial Large Language Models (LLMs) have shown
that non-expert users can jailbreak LLMs by simply manipulating the prompts;
resulting in degenerate output behavior, privacy and security breaches,
offensive outputs, and violations of content regulator policies. Limited formal
studies have been carried out to formalize and analyze these attacks and their
mitigations. We bridge this gap by proposing a formalism and a taxonomy of
known (and possible) jailbreaks. We perform a survey of existing jailbreak
methods and their effectiveness on open-source and commercial LLMs (such as GPT
3.5, OPT, BLOOM, and FLAN-T5-xxl). We further propose a limited set of prompt
guards and discuss their effectiveness against known attack types.

本研究提出了形式主义和已知（和可能的）越狱攻击分类，并在开源和商业 LLM（如 GPT 3.5，OPT，BLOOM 和 FLAN-T5-xxl）上进行了现有越狱方法及其有效性的调查；我们进一步提出了一组有限的提示守卫，并讨论了其对已知攻击类型的有效性。