Structured pruning has been extensively studied on monolingual pre-trained
language models and is yet to be fully evaluated on their multilingual
counterparts. This work investigates three aspects of structured pruning on
multilingual pre-trained language models: settings, algorithms, and efficiency.
Experiments on nine downstream tasks show several counter-intuitive phenomena:
for settings, individually pruning for each language does not induce a better
result; for algorithms, the simplest method performs the best; for efficiency,
a fast model does not imply that it is also small. To facilitate the comparison
on all sparsity levels, we present Dynamic Sparsification, a simple approach
that allows training the model once and adapting to different model sizes at
inference. We hope this work fills the gap in the study of structured pruning
on multilingual pre-trained models and sheds light on future research.

本研究评估了基于结构化剪枝在多语言预训练语言模型上的三个方面：设置、算法和效率，并通过在九个下游任务中的实验展示了一些反直觉现象。为了方便各个稀疏度水平上的比较，我们提出了一种简单的方法 —— 动态稀疏化，它可以让模型只需要训练一次就能适应不同的模型大小。我们希望这项工作填补了关于多语言预训练模型上的结构化剪枝研究的空白，并为未来的研究提供启示。