As the range of applications for Large Language Models (LLMs) continues to
grow, the demand for effective serving solutions becomes increasingly critical.
Despite the versatility of LLMs, no single model can optimally address all
tasks and applications, particularly when balancing performance with cost. This
limitation has led to the development of LLM routing systems, which combine the
strengths of various models to overcome the constraints of individual LLMs.
Yet, the absence of a standardized benchmark for evaluating the performance of
LLM routers hinders progress in this area. To bridge this gap, we present
ROUTERBENCH, a novel evaluation framework designed to systematically assess the
efficacy of LLM routing systems, along with a comprehensive dataset comprising
over 405k inference outcomes from representative LLMs to support the
development of routing strategies. We further propose a theoretical framework
for LLM routing, and deliver a comparative analysis of various routing
approaches through ROUTERBENCH, highlighting their potentials and limitations
within our evaluation framework. This work not only formalizes and advances the
development of LLM routing systems but also sets a standard for their
assessment, paving the way for more accessible and economically viable LLM
deployments. The code and data are available at
this https URL

提供了 ROUTERBENCH 评估框架和数据集，该框架可系统评估 LLM 路由系统的效力并比较各种路由方法的潜力和局限性，进一步推动了 LLM 路由系统的发展和评估标准的确立。

ROUTERBENCH：多层逻辑系统的基准测试

ROUTERBENCH: A Benchmark for Multi-LLM Routing System

Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling
multilingual translation models to billions of parameters without a
proportional increase in training computation. However, MoE models are
prohibitively large and practitioners often resort to methods such as
distillation for serving. In this work, we investigate routing strategies at
different granularity (token, sentence, task) in MoE models to bypass
distillation. Experiments on WMT and a web-scale dataset suggest that
task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy
sub-networks from large sparse models. On WMT, our task-MoE with 32 experts
(533M parameters) outperforms the best performing token-level MoE model
(token-MoE) by +1.0 BLEU on average across 30 language pairs. The peak
inference throughput is also improved by a factor of 1.9x when we route by
tasks instead of tokens. While distilling a token-MoE to a smaller dense model
preserves only 32% of the BLEU gains, our sub-network task-MoE, by design,
preserves all the gains with the same inference cost as the distilled student
model. Finally, when scaling up to 200 language pairs, our 128-expert task-MoE
(13B parameters) performs competitively with a token-level counterpart, while
improving the peak inference throughput by a factor of 2.6x.

该论文研究了不同粒度（令牌、句子、任务）MoE 模型中的路由策略，以绕过蒸馏。通过任务级路由（task-MoE）在 WMT 和 Web 规模数据集上进行的实验表明，我们能够从大型稀疏模型中提取更小、可部署的子网络。对于 30 种语言对，我们的 32 个专家的 task-MOE （533M 个参数）在 WMT 上的表现平均比表现最佳的令牌级 MoE 模型（token-MoE）高 1.0 BLEU。而将一个 token-MoE 蒸馏成较小的密集模型只能保留 32% 的 BLEU 收益，但通过设计，我们的子网络 task-MoE 可以保留所有收益，并且和蒸馏后的 student 模型有相同的推理成本。最后，当扩展到 200 种语言对时，我们的 128 个专家的 task-MoE（13B 个参数）表现与令牌级模型相近，并且提高了推理吞吐量 2.6 倍。

超越蒸馏：面向任务的专家混合方法实现高效推断

Beyond Distillation: Task-level Mixture-of-Experts for Efficient  Inference

Communication of signals among nodes in a complex network poses fundamental
problems of efficiency and cost. Routing of messages along shortest paths
requires global information about the topology, while spreading by diffusion,
which operates according to local topological features, is informationally
"cheap" but inefficient. We introduce a stochastic model for network
communication that combines varying amounts of local and global information
about the network topology. The model generates a continuous spectrum of
dynamics that converge onto shortest-path and random-walk (diffusion)
communication processes at the limiting extremes. We implement the model on two
cohorts of human connectome networks and investigate the effects of varying
amounts of local and global information on the network's communication cost. We
identify routing strategies that approach a (highly efficient) shortest-path
communication process with a relatively small amount of global information.
Moreover, we show that the cost of routing messages from and to hub nodes
varies as a function of the amount of global information driving the system's
dynamics. Finally, we implement the model to identify individual subject
differences from a communication dynamics point of view. The present framework
departs from the classical shortest paths vs. diffusion dichotomy, suggesting
instead that brain networks may exhibit different types of communication
dynamics depending on varying functional demands and the availability of
resources.

本文研究了复杂网络中信号节点之间的通信效率和成本问题，并引入了一种随机模型来结合关于网络拓扑的本地和全局信息，从而产生连续谱的动态，其中包括最短路径和随机游走通信过程。接着，我们在人类连接组网络的两个组别上实现了该模型，并研究了不同程度本地和全局信息对网络通信成本的影响。我们还发现了一些逼近高效的最短路径通信过程的路由策略，并展示了全局信息驱动系统动态时，对中枢节点进行消息路由的成本随全局信息的数量变化而变化。最终，我们使用该模型来从通信动态角度识别个体主体之间的差异，该研究为经典的最短路径与扩散范式之外的大脑网络中存在不同类型的通信动态提供了新的理论支持。