Web user data plays a central role in the ecosystem of pre-trained large
language models (LLMs) and their fine-tuned variants. Billions of data are
crawled from the web and fed to LLMs. How can \textit{\textbf{everyday web
users}} confirm if LLMs misuse their data without permission? In this work, we
suggest that users repeatedly insert personal passphrases into their documents,
enabling LLMs to memorize them. These concealed passphrases in user documents,
referred to as \textit{ghost sentences}, once they are identified in the
generated content of LLMs, users can be sure that their data is used for
training. To explore the effectiveness and usage of this copyrighting tool, we
define the \textit{user training data identification} task with ghost
sentences. Multiple datasets from various sources at different scales are
created and tested with LLMs of different sizes. For evaluation, we introduce a
last $k$ words verification manner along with two metrics: document and user
identification accuracy. In the specific case of instruction tuning of a 3B
LLaMA model, 11 out of 16 users with ghost sentences identify their data within
the generation content. These 16 users contribute 383 examples to $\sim$1.8M
training documents. For continuing pre-training of a 1.1B TinyLlama model, 61
out of 64 users with ghost sentences identify their data within the LLM output.
These 64 users contribute 1156 examples to $\sim$10M training documents.

通过在用户文档中插入隐藏密码句子，可以确认大型语言模型（LLM）是否未经许可滥用用户数据，该研究介绍了一种版权工具，用于识别 LLM 生成内容中的用户数据，通过 ghost sentences 任务定义用户训练数据的识别，并使用文档和用户识别准确率等指标进行评估。