Large pre-trained models have enabled significant advances in machine
learning and served as foundation components. Model fusion methods, such as
task arithmetic, have been proven to be powerful and scalable to incorporate
fine-tuned weights from different tasks into a multi-task model. However,
efficiently fine-tuning large pre-trained models on multiple downstream tasks
remains challenging, leading to inefficient multi-task model fusion. In this
work, we propose a novel method to improve multi-task fusion for
parameter-efficient fine-tuning techniques like LoRA fine-tuning. Specifically,
our approach partially linearizes only the adapter modules and applies task
arithmetic over the linearized adapters. This allows us to leverage the the
advantages of model fusion over linearized fine-tuning, while still performing
fine-tuning and inference efficiently. We demonstrate that our partial
linearization technique enables a more effective fusion of multiple tasks into
a single model, outperforming standard adapter tuning and task arithmetic
alone. Experimental results demonstrate the capabilities of our proposed
partial linearization technique to effectively construct unified multi-task
models via the fusion of fine-tuned task vectors. We evaluate performance over
an increasing number of tasks and find that our approach outperforms standard
parameter-efficient fine-tuning techniques. The results highlight the benefits
of partial linearization for scalable and efficient multi-task model fusion.

提出了一种部分线性化技术，用于改善多任务融合，并通过融合精调的任务向量来构建统一的多任务模型，取得了比标准参数高效微调技术更好的性能。

参数高效的多任务模型融合与部分线性化

Parameter Efficient Multi-task Model Fusion with Partial Linearization

Deep model fusion/merging is an emerging technique that merges the parameters
or predictions of multiple deep learning models into a single one. It combines
the abilities of different models to make up for the biases and errors of a
single model to achieve better performance. However, deep model fusion on
large-scale deep learning models (e.g., LLMs and foundation models) faces
several challenges, including high computational cost, high-dimensional
parameter space, interference between different heterogeneous models, etc.
Although model fusion has attracted widespread attention due to its potential
to solve complex real-world tasks, there is still a lack of complete and
detailed survey research on this technique. Accordingly, in order to understand
the model fusion method better and promote its development, we present a
comprehensive survey to summarize the recent progress. Specifically, we
categorize existing deep model fusion methods as four-fold: (1) "Mode
connectivity", which connects the solutions in weight space via a path of
non-increasing loss, in order to obtain better initialization for model fusion;
(2) "Alignment" matches units between neural networks to create better
conditions for fusion; (3) "Weight average", a classical model fusion method,
averages the weights of multiple models to obtain more accurate results closer
to the optimal solution; (4) "Ensemble learning" combines the outputs of
diverse models, which is a foundational technique for improving the accuracy
and robustness of the final model. In addition, we analyze the challenges faced
by deep model fusion and propose possible research directions for model fusion
in the future. Our review is helpful in deeply understanding the correlation
between different model fusion methods and practical application methods, which
can enlighten the research in the field of deep model fusion.

深度模型融合是将多个深度学习模型的参数或预测合并为一个模型的新兴技术，以提高性能。研究对大规模深度学习模型的融合存在高计算成本、高维参数空间和不同异构模型之间的干扰等挑战。鉴于对该技术缺乏完整详细的调查研究，我们呈现了一份综合调查报告以总结最近的进展，并提出了未来模型融合的研究方向。该综述有助于深入了解不同模型融合方法与实际应用方法之间的关联，从而启发研究深度模型融合领域。

深度模型融合：一项综述

Deep Model Fusion: A Survey

Federated learning is a new learning paradigm that decouples data collection
and model training via multi-party computation and model aggregation. As a
flexible learning setting, federated learning has the potential to integrate
with other learning frameworks. We conduct a focused survey of federated
learning in conjunction with other learning algorithms. Specifically, we
explore various learning algorithms to improve the vanilla federated averaging
algorithm and review model fusion methods such as adaptive aggregation,
regularization, clustered methods, and Bayesian methods. Following the emerging
trends, we also discuss federated learning in the intersection with other
learning paradigms, termed as federated x learning, where x includes multitask
learning, meta-learning, transfer learning, unsupervised learning, and
reinforcement learning. This survey reviews the state of the art, challenges,
and future directions.

在机器学习中，联邦学习是一种新的学习范式，可以通过多方计算和模型聚合来解耦数据采集和模型训练。本文针对联邦学习与其他学习算法相结合的场景进行了调研，并探讨了各种学习算法，以改进平均算法。此外，本文还介绍了模型融合方法和联邦学习与多任务学习、元学习、迁移学习、无监督学习和强化学习相交的领域。综述当前现状、挑战和未来方向。