Large language models (LLMs) have emerged as powerful tools for tackling
complex tasks across diverse domains, but they also raise privacy concerns when
fine-tuned on sensitive data due to potential memorization. While differential
privacy (DP) offers a promising solution by ensuring models are `almost
indistinguishable' with or without any particular privacy unit, current
evaluations on LLMs mostly treat each example (text record) as the privacy
unit. This leads to uneven user privacy guarantees when contributions per user
vary. We therefore study user-level DP motivated by applications where it
necessary to ensure uniform privacy protection across users. We present a
systematic evaluation of user-level DP for LLM fine-tuning on natural language
generation tasks. Focusing on two mechanisms for achieving user-level DP
guarantees, Group Privacy and User-wise DP-SGD, we investigate design choices
like data selection strategies and parameter tuning for the best
privacy-utility tradeoff.

对自然语言生成任务中使用大型语言模型进行用户级差分隐私保护的设计选择进行系统评估，重点研究两种实现用户级差分隐私保证的机制，即群体隐私和用户级 DP-SGD，包括数据选择策略和参数调优，以实现最佳的隐私 - 效用平衡。

用户级别差分隐私用于语言模型微调

Mind the Privacy Unit! User-Level Differential Privacy for Language  Model Fine-Tuning

Aspect category sentiment analysis has attracted increasing research
attention. The dominant methods make use of pre-trained language models by
learning effective aspect category-specific representations, and adding
specific output layers to its pre-trained representation. We consider a more
direct way of making use of pre-trained language models, by casting the ACSA
tasks into natural language generation tasks, using natural language sentences
to represent the output. Our method allows more direct use of pre-trained
knowledge in seq2seq language models by directly following the task setting
during pre-training. Experiments on several benchmarks show that our method
gives the best reported results, having large advantages in few-shot and
zero-shot settings.

本文提出了一种通过将 ACSA 任务转化为自然语言生成任务，使得预训练语言模型可以更直接地应用于该任务设置，从而提高了在各种评估任务中的准确性，特别是在几次学习和零次学习设置下具有显著的优势。

将方面类别情感分析作为一个文本生成任务来解决

Solving Aspect Category Sentiment Analysis as a Text Generation Task

Fine-tuning is the de facto way to leverage large pretrained language models
to perform downstream tasks. However, it modifies all the language model
parameters and therefore necessitates storing a full copy for each task. In
this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning
for natural language generation tasks, which keeps language model parameters
frozen, but optimizes a small continuous task-specific vector (called the
prefix). Prefix-tuning draws inspiration from prompting, allowing subsequent
tokens to attend to this prefix as if it were "virtual tokens". We apply
prefix-tuning to GPT-2 for table-to-text generation and to BART for
summarization. We find that by learning only 0.1\% of the parameters,
prefix-tuning obtains comparable performance in the full data setting,
outperforms fine-tuning in low-data settings, and extrapolates better to
examples with topics unseen during training.

本文提出字首调整 (prefix-tuning)，来解决专用模型使用全量预训练语言模型的空间问题，自动学习小、定制化的向量。通过应用于自然语言生成和文本总结，仅使用 0.1% 的参数即能达到很好的性能。