Large language models (LLMs) often struggle with strict memory, latency, and
power demands. To meet these demands, various forms of dynamic sparsity have
been proposed that reduce compute on an input-by-input basis. These methods
improve over static methods by exploiting the variance across individual
inputs, which has steadily grown with the exponential increase in training
data. Yet, the increasing depth within modern models, currently with hundreds
of layers, has opened opportunities for dynamic layer sparsity, which skips the
computation for entire layers. In this work, we explore the practicality of
layer sparsity by profiling residual connections and establish the relationship
between model depth and layer sparsity. For example, the residual blocks in the
OPT-66B model have a median contribution of 5% to its output. We then take
advantage of this dynamic sparsity and propose Radial Networks, which perform
token-level routing between layers guided by a trained router module. These
networks can be used in a post-training distillation from sequential networks
or trained from scratch to co-learn the router and layer weights. They enable
scaling to larger model sizes by decoupling the number of layers from the
dynamic depth of the network, and their design allows for layer reuse. By
varying the compute token by token, they reduce the overall resources needed
for generating entire sequences. Overall, this leads to larger capacity
networks with significantly lower compute and serving costs for large language
models.

大型语言模型在内存、延迟和能耗方面常常面临困难。本文探讨了层稀疏性的实用性，并描述了一种基于 Token 级路由的径像网络方法，以减少生成整个序列所需的计算资源，实现更大容量的网络并明显降低计算和服务成本。

径向网络：高性能大型语言模型的动态层路由

Radial Networks: Dynamic Layer Routing for High-Performance Large  Language Models

The optimal power flow (OPF) problem determines power generation/demand that
minimize a certain objective such as generation cost or power loss. It is
nonconvex. We prove that, for radial networks, after shrinking its feasible set
slightly, the global optimum of OPF can be recovered via a second-order cone
programming (SOCP) relaxation under a condition that can be checked a priori.
The condition holds for the IEEE 13-, 34-, 37-, 123-bus networks and two
real-world networks, and has a physical interpretation.

通过将可行集合轻微缩小，针对径向网络，可以在先验可检查的条件下通过二阶锥规划 (SOCP) 松弛恢复 OPF 的全局最优解。针对 IEEE 13、34、37、123 节点总线网络和两个实际网络，该条件成立并具有物理解释。