Mixture-of-Experts (MoE) models have shown remarkable capability in
instruction tuning, especially when the number of tasks scales. However,
previous methods simply merge all training tasks (e.g. creative writing,
coding, and mathematics) and apply fixed sampling weights, without considering
the importance of different tasks as the model training state changes. In this
way, the most helpful data cannot be effectively distinguished, leading to
suboptimal model performance. To reduce the potential redundancies of datasets,
we make the first attempt and propose a novel dynamic data mixture for MoE
instruction tuning. Specifically, inspired by MoE's token routing preference,
we build dataset-level representations and then capture the subtle differences
among datasets. Finally, we propose to dynamically adjust the sampling weight
of datasets by their inter-redundancies, thus maximizing global performance
under a limited training budget. The experimental results on two MoE models
demonstrate the effectiveness of our approach on both downstream knowledge \&
reasoning tasks and open-ended queries. Code and models are available at
this https URL .

基于混合专家模型（Mixture-of-Experts），提出了一种动态数据混合的处理方法以优化模型性能，通过动态地调整训练数据的采样权重，减少数据集中的冗余，从而在有限的训练预算下最大化整体性能。