While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers their efficiency and diversity. To address these challenges, we draw inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse mixture-of-agents (SMoA) framework to improve the efficiency and diversity of multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel Response Selection and Early Stopping mechanisms to sparsify information flows among individual LLM agents, striking a balance between performance and efficiency. Additionally, inspired by the expert diversity principle in SMoE frameworks for workload balance between experts, we assign distinct role descriptions to each LLM agent, fostering diverse and divergent thinking. Extensive experiments on reasoning, alignment, and fairness benchmarks demonstrate that SMoA achieves performance comparable to traditional mixture-of-agents approaches but with significantly lower computational costs. Further analysis reveals that SMoA is more stable, has a greater capacity to scale, and offers considerable potential through hyper-parameter optimization. Code and data will be available at: https://github.com/David-Li0406/SMoA.

本文针对多代理系统在大语言模型（LLMs）中面临的效率和多样性问题，提出了一种稀疏混合代理（SMoA）框架。该框架通过引入响应选择和提前停止机制，优化了代理间的信息流动，从而在性能与效率之间取得平衡。研究结果表明，SMoA在较低的计算成本下，能够实现与传统混合代理方法相媲美的性能，同时具备更高的稳定性和可扩展性。

SMoA：通过稀疏混合代理改进多代理大语言模型