Prompt leakage in large language models (LLMs) poses a significant security
and privacy threat, particularly in retrieval-augmented generation (RAG)
systems. However, leakage in multi-turn LLM interactions along with mitigation
strategies has not been studied in a standardized manner. This paper
investigates LLM vulnerabilities against prompt leakage across 4 diverse
domains and 10 closed- and open-source LLMs. Our unique multi-turn threat model
leverages the LLM's sycophancy effect and our analysis dissects task
instruction and knowledge leakage in the LLM response. In a multi-turn setting,
our threat model elevates the average attack success rate (ASR) to 86.2%,
including a 99% leakage with GPT-4 and claude-1.3. We find that some black-box
LLMs like Gemini show variable susceptibility to leakage across domains - they
are more likely to leak contextual knowledge in the news domain compared to the
medical domain. Our experiments measure specific effects of 6 black-box defense
strategies, including a query-rewriter in the RAG scenario. Our proposed
multi-tier combination of defenses still has an ASR of 5.3% for black-box LLMs,
indicating room for enhancement and future direction for LLM security research.

对大型语言模型（LLMs）中的提示泄漏进行了研究，发现在多轮 LLM 交互中存在漏洞和泄漏，并提出了防御策略。

研究多轮 LLM 交互的提示泄露效应和黑盒防御

Investigating the prompt leakage effect and black-box defenses for  multi-turn LLM interactions

Prompt, recognized as crucial intellectual property, enables large language
models (LLMs) to perform specific tasks without the need of fine-tuning,
underscoring their escalating importance. With the rise of prompt-based
services, such as prompt marketplaces and LLM applications, providers often
display prompts' capabilities through input-output examples to attract users.
However, this paradigm raises a pivotal security concern: does the exposure of
input-output pairs pose the risk of potential prompt leakage, infringing on the
intellectual property rights of the developers? To our knowledge, this problem
still has not been comprehensively explored yet. To remedy this gap, in this
paper, we perform the first in depth exploration and propose a novel attack
framework for reverse-stealing prompts against commercial LLMs, namely PRSA.
The main idea of PRSA is that by analyzing the critical features of the
input-output pairs, we mimic and gradually infer (steal) the target prompts. In
detail, PRSA mainly consists of two key phases: prompt mutation and prompt
pruning. In the mutation phase, we propose a prompt attention algorithm based
on differential feedback to capture these critical features for effectively
inferring the target prompts. In the prompt pruning phase, we identify and mask
the words dependent on specific inputs, enabling the prompts to accommodate
diverse inputs for generalization. Through extensive evaluation, we verify that
PRSA poses a severe threat in real world scenarios. We have reported these
findings to prompt service providers and actively collaborate with them to take
protective measures for prompt copyright.

通过分析输入 - 输出对的关键特征，在商业 LLM 中针对 prompt 反向窃取设计了一种新的攻击框架，名为 PRSA，从而构成了一个严重的潜在威胁。