Large language models are shown to present privacy risks through memorization
of training data, and several recent works have studied such risks for the
pre-training phase. Little attention, however, has been given to the
fine-tuning phase and it is not well understood how different fine-tuning
methods (such as fine-tuning the full model, the model head, and adapter)
compare in terms of memorization risk. This presents increasing concern as the
"pre-train and fine-tune" paradigm proliferates. In this paper, we empirically
study memorization of fine-tuning methods using membership inference and
extraction attacks, and show that their susceptibility to attacks is very
different. We observe that fine-tuning the head of the model has the highest
susceptibility to attacks, whereas fine-tuning smaller adapters appears to be
less vulnerable to known extraction attacks.

本论文通过成员推断和提取攻击实证研究了不同调参方法（如全模型微调、模型头微调和适配器微调）的记忆化风险，发现微调头部的风险最高，而微调较小的适配器则较不容易受到已知提取攻击的影响。