Recent work has proposed explicitly inducing language-wise modularity in
multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a
means of better guiding cross-lingual sharing. In this work, we investigate (1)
the degree to which language-wise modularity naturally arises within models
with no special modularity interventions, and (2) how cross-lingual sharing and
interference differ between such models and those with explicit SFT-guided
subnetwork modularity. To quantify language specialization and cross-lingual
interaction, we use a Training Data Attribution method that estimates the
degree to which a model's predictions are influenced by in-language or
cross-language training examples. Our results show that language-specialized
subnetworks do naturally arise, and that SFT, rather than always increasing
modularity, can decrease language specialization of subnetworks in favor of
more cross-lingual sharing.

通过使用稀疏微调的语言子网络作为更好引导跨语言共享的手段，最近的研究提出在多语言语言模型中明确引入语言模块化。本文研究了 (1) 在没有特殊模块化干预的情况下，语言模块化自然形成的程度，以及 (2) 这种模型与具有显式稀疏微调引导的子网络模块化的模型之间跨语言共享和干扰的差异。通过使用训练数据归因方法来量化语言专门化和跨语言交互，我们的结果表明，语言专门化的子网络确实自然形成，并且稀疏微调不一定增加模块化，而是可能减少子网络的语言专门化，更倾向于进行跨语言共享。