Large language models (LLMs) can memorize many pretrained sequences verbatim.
This paper studies if we can locate a small set of neurons in LLMs responsible
for memorizing a given sequence. While the concept of localization is often
mentioned in prior work, methods for localization have never been
systematically and directly evaluated; we address this with two benchmarking
approaches. In our INJ Benchmark, we actively inject a piece of new information
into a small subset of LLM weights and measure whether localization methods can
identify these "ground truth" weights. In the DEL Benchmark, we study
localization of pretrained data that LLMs have already memorized; while this
setting lacks ground truth, we can still evaluate localization by measuring
whether dropping out located neurons erases a memorized sequence from the
model. We evaluate five localization methods on our two benchmarks, and both
show similar rankings. All methods exhibit promising localization ability,
especially for pruning-based methods, though the neurons they identify are not
necessarily specific to a single memorized sequence.

本研究主要研究在大型语言模型中是否可以确定负责记忆给定序列的少量神经元，并通过两种基准方法进行了评估。评估结果显示，尽管所识别的神经元不一定与单个记忆序列相关，但所有方法都展现出令人满意的定位能力，特别是基于修剪的方法。

LLM 中的本地化方法是否真正本地化记忆数据？

Do Localization Methods Actually Localize Memorized Data in LLMs?

Past work has shown that large language models are susceptible to privacy
attacks, where adversaries generate sequences from a trained model and detect
which sequences are memorized from the training set. In this work, we show that
the success of these attacks is largely due to duplication in commonly used
web-scraped training sets. We first show that the rate at which language models
regenerate training sequences is superlinearly related to a sequence's count in
the training set. For instance, a sequence that is present 10 times in the
training data is on average generated ~1000 times more often than a sequence
that is present only once. We next show that existing methods for detecting
memorized sequences have near-chance accuracy on non-duplicated training
sequences. Finally, we find that after applying methods to deduplicate training
data, language models are considerably more secure against these types of
privacy attacks. Taken together, our results motivate an increased focus on
deduplication in privacy-sensitive applications and a reevaluation of the
practicality of existing privacy attacks.

该研究显示，大型语言模型在面对隐私攻击时，其攻击的成功与常用网络爬取的训练集中的重复数据有很大关系，而消除重复数据的方法可以显著提高语言模型的隐私安全性。