Averaging the parameters of models that have the same architecture and
initialization can provide a means of combining their respective capabilities.
In this paper, we take the perspective that this "merging" operation can be
seen as choosing parameters that approximately maximize the joint likelihood of
the posteriors of the models' parameters. Computing a simple average of the
models' parameters therefore corresponds to making an isotropic Gaussian
approximation to their posteriors. We develop an alternative merging procedure
based on the Laplace approximation where we approximate each model's posterior
as a Gaussian distribution whose precision matrix corresponds to its Fisher
information. We first show that our "Fisher merging" technique provides a
performance boost in settings where simple parameter averaging is currently
used -- specifically, robust fine-tuning and model ensembling. Then, we compare
merging to standard gradient-based transfer learning and demonstrate that
merging enables a fundamentally different method for transferring capabilities
across models. Specifically, we show that Fisher merging is competitive with
gradient-based transfer learning approaches (while being significantly cheaper)
in intermediate-task training and domain-adaptive pre-training. We also show
that our merging procedure makes it possible to combine models in previously
unexplored ways. We release our code to facilitate future research into methods
for merging models.

本研究提出了 Fisher 合并方法用于模型合并，与标准梯度传递学习方法相比，Fisher 合并可以在中间任务训练和领域自适配预训练方面提供一种基本不同的能力转移方法，同时成本更低，并且同时能够实现以前未曾探索的模型组合方式。