In the era of large language models, model merging is a promising way to
combine multiple task-specific models into a single multitask model without
extra training. However, two challenges remain: (a) interference between
different models and (b) heterogeneous data during testing. Traditional model
merging methods often show significant performance gaps compared to fine-tuned
models due to these issues. Additionally, a one-size-fits-all model lacks
flexibility for diverse test data, leading to performance degradation. We show
that both shared and exclusive task-specific knowledge are crucial for merging
performance, but directly merging exclusive knowledge hinders overall
performance. In view of this, we propose Twin-Merging, a method that
encompasses two principal stages: (1) modularizing knowledge into shared and
exclusive components, with compression to reduce redundancy and enhance
efficiency; (2) dynamically merging shared and task-specific knowledge based on
the input. This approach narrows the performance gap between merged and
fine-tuned models and improves adaptability to heterogeneous data. Extensive
experiments on $12$ datasets for both discriminative and generative tasks
demonstrate the effectiveness of our method, showing an average improvement of
$28.34\%$ in absolute normalized score for discriminative tasks and even
surpassing the fine-tuned upper bound on the generative tasks. (Our
implementation is available in this https URL)

在大型语言模型时代，模型合并是将多个特定任务模型合并为一个多任务模型的有希望的方法，但面临着不同模型之间的干扰和测试期间的异构数据等两个挑战。我们提出了 Twin-Merging 方法，它模块化知识为共享和专属组件，并在输入的基础上动态地合并共享和任务特定的知识，从而缩小了合并模型和微调模型之间的性能差距，并提高了对异构数据的适应性。广泛的实验表明了我们方法的有效性，对于判别性任务平均标准化得分提升了 28.34%，在生成性任务上甚至超过了微调模型的上限。