With the proliferation of domain-specific models, model merging has emerged
as a set of techniques that combine the capabilities of multiple models into
one that can multitask without the cost of additional training. In this paper,
we propose a new model merging technique, Drop and rEscaLe via sampLing with
mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
parameters in order of their magnitude and assigns higher dropout probabilities
(p) to parameters with lower ranks corresponding to lower magnitudes. To
approximate the original embeddings, MAGPRUNE employs a rescaling operation on
the parameters that survive the random dropping by 1/(1 - p). On three
different expert models considered for merging (LM, Math, Code) and
corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
average improvement of 2.4 points over baseline methods employing delta
parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
code at: this https URL

利用一种新的模型合并技术 DELLA-Merging，它采用了一种名为 MAGPRUNE 的修剪技术，通过首先按照参数的大小对其进行排名并给较小的参数分配更高的丢弃概率 (p)，接着在随机丢弃的参数上通过缩放操作近似原始嵌入。在三种不同的专家模型和相应的基准数据集上，DELLA 相较于基线方法（delta 参数修剪）平均提升了 2.4 个点（相较于 TIES 提升了 3.6 个点，相较于 DARE 提升了 1.2 个点），相较于无修剪的基线方法（TA）提升了 11.1 个点。