In this work, we propose a method that combines two popular research areas by
injecting linguistic structures into pre-trained language models in the
parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel
adapter modules encoding different linguistic structures are combined using a
novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates
are used to determine the importance of these modules at each layer of the
model. To reduce the number of parameters, we first train the model for a fixed
small number of steps before pruning the experts based on their importance
scores. Our experiment results with three different pre-trained models show
that our approach can outperform state-of-the-art PEFT methods with a
comparable number of parameters. In addition, we provide additional analysis to
examine the experts selected by each model at each layer to provide insights
for future studies.

本研究提出了一种方法，在参数高效微调（PEFT）环境中将语言结构注入到预训练语言模型中。我们使用一种新颖的语言专家混合体架构，将编码不同语言结构的并行适配器模块进行组合，并使用 Gumbel-Softmax 门来确定模型每一层中这些模块的重要性。为了减少参数数量，在修剪专家之前，我们首先对模型进行了固定步骤的训练。我们用三种不同的预训练模型进行实验，结果表明我们的方法在具有相当数量的参数的情况下，能够胜过最先进的 PEFT 方法。此外，我们还提供了额外分析，以检查和提供未来研究的见解。