The rapid advancement of large language models (LLMs) has led to
architectures with billions to trillions of parameters, posing significant
deployment challenges due to their substantial demands on memory, processing
power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures
have emerged as a solution, activating only a subset of parameters per token,
thereby achieving faster inference while maintaining performance. However, SMoE
models still face limitations in broader deployment due to their large
parameter counts and significant GPU memory requirements. In this work, we
introduce a gradient-free evolutionary strategy named EEP (Efficient Expert
P}runing) to enhance the pruning of experts in SMoE models. EEP relies solely
on model inference (i.e., no gradient computation) and achieves greater
sparsity while maintaining or even improving performance on downstream tasks.
EEP can be used to reduce both the total number of experts (thus saving GPU
memory) and the number of active experts (thus accelerating inference). For
example, we demonstrate that pruning up to 75% of experts in Mixtral
$8\times7$B-Instruct results in a substantial reduction in parameters with
minimal performance loss. Remarkably, we observe improved performance on
certain tasks, such as a significant increase in accuracy on the SQuAD dataset
(from 53.4% to 75.4%), when pruning half of the experts. With these results,
EEP not only lowers the barrier to deploying SMoE models,but also challenges
the conventional understanding of model pruning by showing that fewer experts
can lead to better task-specific performance without any fine-tuning. Code is
available at this https URL

在本研究中，我们引入了一种名为 EEP（Efficient Expert Pruning）的无梯度进化策略，通过对神经网络模型中的专家进行剪枝，实现更高的稀疏性并在下游任务中保持或提高性能，从而解决大型语言模型的部署问题。