Training a unified multilingual model promotes knowledge transfer but
inevitably introduces negative interference. Language-specific modeling methods
show promise in reducing interference. However, they often rely on heuristics
to distribute capacity and struggle to foster cross-lingual transfer via
isolated modules. In this paper, we explore intrinsic task modularity within
multilingual networks and leverage these observations to circumvent
interference under multilingual translation. We show that neurons in the
feed-forward layers tend to be activated in a language-specific manner.
Meanwhile, these specialized neurons exhibit structural overlaps that reflect
language proximity, which progress across layers. Based on these findings, we
propose Neuron Specialization, an approach that identifies specialized neurons
to modularize feed-forward layers and then continuously updates them through
sparse networks. Extensive experiments show that our approach achieves
consistent performance gains over strong baselines with additional analyses
demonstrating reduced interference and increased knowledge transfer.

通过神经元特化，提出一种解决多语言翻译中干扰和促进知识转移的方法，并通过大量实验证明了其相比强基准模型的性能提升以及减少干扰并增加知识转移的效果。

神经元专业化：利用内在任务模块化实现多语言机器翻译

Neuron Specialization: Leveraging intrinsic task modularity for  multilingual machine translation

This work examines the presence of modularity in pre-trained Transformers, a
feature commonly found in human brains and thought to be vital for general
intelligence. In analogy to human brains, we consider two main characteristics
of modularity: (1) functional specialization of neurons: we evaluate whether
each neuron is mainly specialized in a certain function, and find that the
answer is yes. (2) function-based neuron grouping: we explore finding a
structure that groups neurons into modules by function, and each module works
for its corresponding function. Given the enormous amount of possible
structures, we focus on Mixture-of-Experts as a promising candidate, which
partitions neurons into experts and usually activates different experts for
different inputs. Experimental results show that there are functional experts,
where clustered are the neurons specialized in a certain function. Moreover,
perturbing the activations of functional experts significantly affects the
corresponding function. Finally, we study how modularity emerges during
pre-training, and find that the modular structure is stabilized at the early
stage, which is faster than neuron stabilization. It suggests that Transformers
first construct the modular structure and then learn fine-grained neuron
functions. Our code and data are available at
this https URL

本文研究使用预训练的 Transformers 中是否存在模块化，发现神经元具有功能专业化，可以通过 Mixture-of-Experts 将分组神经元分配到对应功能模块中。在预训练阶段，模块化结构会更快地稳定，文献代码和数据可在链接中获得。