Deep model fusion is an emerging technique that unifies the predictions or
parameters of several deep neural networks into a single model in a
cost-effective and data-efficient manner. This enables the unified model to
take advantage of the original models' strengths, potentially exceeding their
performance. Although a variety of deep model fusion techniques have been
introduced, their evaluations tend to be inconsistent and often inadequate to
validate their effectiveness and robustness against distribution shifts. To
address this issue, we introduce FusionBench, which is the first comprehensive
benchmark dedicated to deep model fusion. FusionBench covers a wide range of
tasks, including open-vocabulary image classification, text classification, and
text-to-text generation. Each category includes up to eight tasks with
corresponding task-specific models, featuring both full fine-tuning and LoRA
fine-tuning, as well as models of different sizes, to ensure fair and balanced
comparisons of various multi-task model fusion techniques across different
tasks, model scales, and fine-tuning strategies. We implement and evaluate a
broad spectrum of deep model fusion techniques. These techniques range from
model ensemble methods, which combine the predictions to improve the overall
performance, to model merging, which integrates different models into a single
one, and model mixing methods, which upscale or recombine the components of the
original models. FusionBench now contains 26 distinct tasks, 74 fine-tuned
models, and 16 fusion techniques, and we are committed to consistently
expanding the benchmark with more tasks, models, and fusion techniques. In
addition, we offer a well-documented set of resources and guidelines to aid
researchers in understanding and replicating the benchmark results. Homepage
this https URL

深度模型融合是一种新兴技术，它以经济高效的方式将多个深度神经网络的预测或参数统一到一个模型中，从而优于原模型的性能。为了解决评估这些融合技术有效性和稳健性的问题，我们介绍了 FusionBench 作为第一个全面专门面向深度模型融合的基准测试。FusionBench 覆盖了各种任务，并在不同任务、模型规模和微调策略下提供了公平且全面的比较。同时，我们还提供了资源和指南，以帮助研究人员理解和复现基准测试结果。

FusionBench 深度模型融合的全面基准

FusionBench: A Comprehensive Benchmark of Deep Model Fusion

Deep model fusion/merging is an emerging technique that merges the parameters
or predictions of multiple deep learning models into a single one. It combines
the abilities of different models to make up for the biases and errors of a
single model to achieve better performance. However, deep model fusion on
large-scale deep learning models (e.g., LLMs and foundation models) faces
several challenges, including high computational cost, high-dimensional
parameter space, interference between different heterogeneous models, etc.
Although model fusion has attracted widespread attention due to its potential
to solve complex real-world tasks, there is still a lack of complete and
detailed survey research on this technique. Accordingly, in order to understand
the model fusion method better and promote its development, we present a
comprehensive survey to summarize the recent progress. Specifically, we
categorize existing deep model fusion methods as four-fold: (1) "Mode
connectivity", which connects the solutions in weight space via a path of
non-increasing loss, in order to obtain better initialization for model fusion;
(2) "Alignment" matches units between neural networks to create better
conditions for fusion; (3) "Weight average", a classical model fusion method,
averages the weights of multiple models to obtain more accurate results closer
to the optimal solution; (4) "Ensemble learning" combines the outputs of
diverse models, which is a foundational technique for improving the accuracy
and robustness of the final model. In addition, we analyze the challenges faced
by deep model fusion and propose possible research directions for model fusion
in the future. Our review is helpful in deeply understanding the correlation
between different model fusion methods and practical application methods, which
can enlighten the research in the field of deep model fusion.

深度模型融合是将多个深度学习模型的参数或预测合并为一个模型的新兴技术，以提高性能。研究对大规模深度学习模型的融合存在高计算成本、高维参数空间和不同异构模型之间的干扰等挑战。鉴于对该技术缺乏完整详细的调查研究，我们呈现了一份综合调查报告以总结最近的进展，并提出了未来模型融合的研究方向。该综述有助于深入了解不同模型融合方法与实际应用方法之间的关联，从而启发研究深度模型融合领域。