Multi-modal large language models (LLM) have achieved powerful capabilities
for visual semantic understanding in recent years. However, little is known
about how LLMs comprehend visual information and interpret different modalities
of features. In this paper, we propose a new method for identifying multi-modal
neurons in transformer-based multi-modal LLMs. Through a series of experiments,
We highlight three critical properties of multi-modal neurons by four
well-designed quantitative evaluation metrics. Furthermore, we introduce a
knowledge editing method based on the identified multi-modal neurons, for
modifying a specific token to another designative token. We hope our findings
can inspire further explanatory researches on understanding mechanisms of
multi-modal LLMs.

通过一系列实验，我们提出了一种基于多模态神经元的新方法，用于识别基于变压器的多模态大型语言模型中的多模态神经元，并通过四个精心设计的定量评估指标突出了多模态神经元的三个关键属性。此外，我们介绍了一种基于识别的多模态神经元的知识编辑方法，用于将一个特定标记修改为另一个指定标记。我们希望我们的发现能够激发对多模态大型语言模型理解机制的进一步解释性研究。