Many recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks to obtain a single multi-task model. Most existing works tackle the simpler setup of merging NNs initialized from a common pre-trained network, where simple heuristics like weight averaging work well. This work targets a more challenging goal: merging large transformers trained on different tasks from distinct initializations. First, we demonstrate that traditional merging methods fail catastrophically in this setup. To overcome this challenge, we propose Foldable SuperNet Merge (FS-Merge), a method that optimizes a SuperNet to fuse the original models using a feature reconstruction loss. FS-Merge is simple, data-efficient, and capable of merging models of varying widths. We test FS-Merge against existing methods, including knowledge distillation, on MLPs and transformers across various settings, sizes, tasks, and modalities. FS-Merge consistently outperforms them, achieving SOTA results, particularly in limited data scenarios.

本研究解决了如何有效合并具有不同初始化和任务的大型变换器模型的问题，通过提出可折叠超网络合并（FS-Merge）方法，优化模型融合过程。FS-Merge 方法在多个设置、任务和模态下表现优异，特别是在数据有限的情况下，显著超越传统合并技术和知识蒸馏方法，展示出其强大的数据效率和适应性。

可折叠超网络：可扩展地合并具有不同初始化和任务的变换器