The application of mixture-of-experts (MoE) is gaining popularity due to its
ability to improve model's performance. In an MoE structure, the gate layer
plays a significant role in distinguishing and routing input features to
different experts. This enables each expert to specialize in processing their
corresponding sub-tasks. However, the gate's routing mechanism also gives rise
to narrow vision: the individual MoE's expert fails to use more samples in
learning the allocated sub-task, which in turn limits the MoE to further
improve its generalization ability. To effectively address this, we propose a
method called Mixture-of-Distilled-Expert (MoDE), which applies moderate mutual
distillation among experts to enable each expert to pick up more features
learned by other experts and gain more accurate perceptions on their original
allocated sub-tasks. We conduct plenty experiments including tabular, NLP and
CV datasets, which shows MoDE's effectiveness, universality and robustness.
Furthermore, we develop a parallel study through innovatively constructing
"expert probing", to experimentally prove why MoDE works: moderate distilling
knowledge can improve each individual expert's test performances on their
assigned tasks, leading to MoE's overall performance improvement.

我们提出了一种叫作 Mixture-of-Distilled-Expert（MoDE）的方法，通过在专家之间应用适度的相互蒸馏，使每个专家能够掌握其他专家学习到的更多特征，从而对其原始分配的子任务有更准确的认识。我们进行了大量的实验，包括表格、自然语言处理和计算机视觉数据集，证明了 MoDE 的有效性、普适性和鲁棒性。此外，我们通过创新地构建 “专家探针” 进行了平行研究，实验性地证明了为什么 MoDE 起作用：适度的知识蒸馏可以提高每个单独专家在其所分配任务上的测试性能，从而提升 MoE 的整体性能。