Fine-tuning pre-trained models provides significant advantages in downstream
performance. The ubiquitous nature of pre-trained models such as BERT and its
derivatives in natural language processing has also led to a proliferation of
task-specific fine-tuned models. As these models typically only perform one
task well, additional training or ensembling is required in multi-task
scenarios. The growing field of model merging provides a solution, dealing with
the challenge of combining multiple task-specific models into a single
multi-task model. In this study, we introduce a novel model merging method for
Transformers, combining insights from previous work in Fisher-weighted
averaging and the use of Fisher information in model pruning. Utilizing the
Fisher information of mask nodes within the Transformer architecture, we devise
a computationally efficient weighted-averaging scheme. Our method exhibits a
regular and significant performance increase across various models in the BERT
family, outperforming full-scale Fisher-weighted averaging in a fraction of the
computational cost, with baseline performance improvements of up to +6.5 and a
speedup of 57.4x. Our results prove the potential of our method in current
multi-task learning environments and suggest its scalability and adaptability
to new model architectures and learning scenarios.

我们引入一种新的模型合并方法，将先前在 Fisher 加权平均和模型修剪中对 Fisher 信息的使用的工作见解与 Transformer 结构内的掩码节点的 Fisher 信息相结合，提出了一种计算高效的加权平均方案。我们的方法在 BERT 系列的各种模型中展现了规则且显著的性能提升，相较于计算成本的全尺寸 Fisher 加权平均，基准性能提升高达 6.5 个百分点，并且以 57.4 倍的加速比提升效率。我们的结果证明了我们的方法在当前多任务学习环境中的潜力，并表明其在新的模型架构和学习场景中的可扩展性和适应性。