We introduce XFT, a simple yet powerful training scheme, by simply merging
upcycled Mixture-of-Experts (MoE) to unleash the performance limit of
instruction-tuned code Large Language Models (LLMs). While vanilla sparse
upcycling fails to improve instruction tuning, XFT introduces a shared expert
mechanism with a novel routing weight normalization strategy into sparse
upcycling, which significantly boosts instruction tuning. After fine-tuning the
upcycled MoE model, XFT introduces a learnable model merging mechanism to
compile the upcycled MoE model back to a dense model, achieving upcycled
MoE-level performance with only dense-model compute. By applying XFT to a 1.3B
model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6
pass@1 on HumanEval and HumanEval+ respectively. With the same data and model
architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+,
along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and
DS-1000, demonstrating its generalizability. XFT is fully orthogonal to
existing techniques such as Evol-Instruct and OSS-Instruct, opening a new
dimension for improving code instruction tuning. Codes are available at
this https URL .

通过将升级 Mixture-of-Experts (MoE) 与稀疏升级相结合，并引入共享专家机制和路由权重归一化策略，XFT 训练方案显著增强了指令调优，将升级 MoE 型号编译回密集型型号，实现了与升级 MoE 型号相当的性能。应用 XFT 到 1.3B 模型，我们创造了一种新的最先进的微型代码 LLM (<3B)，在 HumanEval 和 HumanEval+ 上分别达到了 67.1 和 64.6 的 pass@1。与现有技术如 Evol-Instruct 和 OSS-Instruct 完全不冲突，为改善代码指令调优提供了新的维度。