We present Self-MoE, an approach that transforms a monolithic LLM into a
compositional, modular system of self-specialized experts, named MiXSE (MiXture
of Self-specialized Experts). Our approach leverages self-specialization, which
constructs expert modules using self-generated synthetic data, each equipped
with a shared base LLM and incorporating self-optimized routing. This allows
for dynamic and capability-specific handling of various target tasks, enhancing
overall capabilities, without extensive human-labeled data and added
parameters. Our empirical results reveal that specializing LLMs may exhibit
potential trade-offs in performances on non-specialized tasks. On the other
hand, our Self-MoE demonstrates substantial improvements over the base LLM
across diverse benchmarks such as knowledge, reasoning, math, and coding. It
also consistently outperforms other methods, including instance merging and
weight merging, while offering better flexibility and interpretability by
design with semantic experts and routing. Our findings highlight the critical
role of modularity and the potential of self-improvement in achieving
efficient, scalable, and adaptable systems.

我们提出了一种名为 Self-MoE 的方法，将一个整体的 LLM 转化为由自身专业化的专家模块组成的组合式模块系统 MiXSE。我们的方法利用自我专业化，使用自生成的合成数据构建专家模块，每个模块都配备了共享的基础 LLM，并融入了自我优化的路由。这样可以动态和能力特定地处理各种目标任务，提高整体能力，而无需大量人工标记的数据和额外的参数。我们的实证结果显示，专门化的 LLMs 在非专门化任务上可能存在性能的潜在权衡。另一方面，我们的 Self-MoE 在知识、推理、数学和编码等不同基准测试中，表现出明显的改进。它还在设计上通过语义专家和路由提供更好的灵活性和可解释性。我们的发现突出了模块化和自我改进在实现高效、可扩展和适应性强的系统中的关键作用。