Large Language Models (LLMs) possess vast amounts of knowledge within their
parameters, prompting research into methods for locating and editing this
knowledge. Previous investigations have primarily focused on fill-in-the-blank
tasks and locating entity-related usually single-token facts) information in
relatively small-scale language models. However, several key questions remain
unanswered: (1) How can we effectively locate query-relevant neurons in
contemporary autoregressive LLMs, such as LLaMA and Mistral? (2) How can we
address the challenge of long-form text generation? (3) Are there localized
knowledge regions in LLMs? In this study, we introduce Neuron
Attribution-Inverse Cluster Attribution (NA-ICA), a novel architecture-agnostic
framework capable of identifying key neurons in LLMs. NA-ICA allows for the
examination of long-form answers beyond single tokens by employing the proxy
task of multi-choice question answering. To evaluate the effectiveness of our
detected key neurons, we construct two multi-choice QA datasets spanning
diverse domains and languages. Empirical evaluations demonstrate that NA-ICA
outperforms baseline methods significantly. Moreover, analysis of neuron
distributions reveals the presence of visible localized regions, particularly
within different domains. Finally, we demonstrate the potential applications of
our detected key neurons in knowledge editing and neuron-based prediction.

本研究引入了神经元归因逆向集群归因（NA-ICA）的新架构不受限于特定模型，能够有效识别大型语言模型中的关键神经元，并通过多选题回答的代理任务检查单词以外的长文本答案。经验评估证明，NA-ICA 明显优于基准方法。此外，对神经元分布的分析揭示了可见的局部区域，尤其在不同领域之间。最后，我们展示了关键神经元在知识编辑和基于神经元的预测中的潜在应用。