Projecting visual features into word embedding space has become a significant
fusion strategy adopted by Multimodal Large Language Models (MLLMs). However,
its internal mechanisms have yet to be explored. Inspired by multilingual
research, we identify domain-specific neurons in multimodal large language
models. Specifically, we investigate the distribution of domain-specific
neurons and the mechanism of how MLLMs process features from diverse domains.
Furthermore, we propose a three-stage framework for language model modules in
MLLMs when handling projected image features, and verify this hypothesis using
logit lens. Extensive experiments indicate that while current MLLMs exhibit
Visual Question Answering (VQA) capability, they may not fully utilize
domain-specific information. Manipulating domain-specific neurons properly will
result in a 10\% change of accuracy at most, shedding light on the development
of cross-domain, all-encompassing MLLMs in the future. Our code will be
released upon paper notification.

通过研究多模态大型语言模型的内部机制，我们在多语种研究的启发下，发现了多模态大型语言模型中的领域特定神经元，并研究了这些模型如何处理来自不同领域的特征。进一步地，我们提出了一个三阶段的语言模型模块框架，用于处理投影的图像特征，并使用逻辑镜头验证了这一假设。大量实验证明，虽然当前的多模态大型语言模型具备视觉问答的能力，但它们可能没有充分利用领域特定的信息。适当地操纵领域特定的神经元最多会导致 10% 的准确率变化，为未来跨领域、全方位的多模态大型语言模型的发展提供了启示。我们的代码将在论文通知后发布。