Fine-tuning pre-trained language models has become the prevalent paradigm for
building downstream NLP models. Oftentimes fine-tuned models are readily
available but their training data is not, due to data privacy or intellectual
property concerns. This creates a barrier to fusing knowledge across individual
models to yield a better single model. In this paper, we study the problem of
merging individual models built on different training data sets to obtain a
single model that performs well both across all data set domains and can
generalize on out-of-domain data. We propose a dataless knowledge fusion method
that merges models in their parameter space, guided by weights that minimize
prediction differences between the merged model and the individual models. Over
a battery of evaluation settings, we show that the proposed method
significantly outperforms baselines such as Fisher-weighted averaging or model
ensembling. Further, we find that our method is a promising alternative to
multi-task learning that can preserve or sometimes improve over the individual
models without access to the training data. Finally, model merging is more
efficient than training a multi-task model, thus making it applicable to a
wider set of scenarios.

本文提出一种数据无关的知识融合方法，通过在参数空间内合并模型并引导最小化预测差异的权重，将建立在不同训练数据集上的个别模型合并为一个模型，以在所有数据集领域都表现良好并可以推广到域外数据。 在评估设置的一系列电池上，我们发现该方法明显优于 Fisher 加权平均或模型集成等基线，并且我们发现我们的方法是一种有前途的替代多任务学习的方法，可以在不访问训练数据的情况下保留或有时提高个别模型的性能。最后，模型合并比训练多任务模型更高效，因此适用于更广泛的情况。