We show how to improve the inference efficiency of an LLM by expanding it
into a mixture of sparse experts, where each expert is a copy of the original
weights, one-shot pruned for a specific cluster of input values. We call this
approach $\textit{Sparse Expansion}$. We show that, for models such as Llama 2
70B, as we increase the number of sparse experts, Sparse Expansion outperforms
all other one-shot sparsification approaches for the same inference FLOP budget
per token, and that this gap grows as sparsity increases, leading to inference
speedups.
But why? To answer this, we provide strong evidence that the mixture of
sparse experts is effectively $\textit{disentangling}$ the input-output
relationship of every individual neuron across clusters of inputs.
Specifically, sparse experts approximate the dense neuron output distribution
with fewer weights by decomposing the distribution into a collection of simpler
ones, each with a separate sparse dot product covering it. Interestingly, we
show that the Wasserstein distance between a neuron's output distribution and a
Gaussian distribution is an indicator of its entanglement level and
contribution to the accuracy of the model. Every layer of an LLM has a fraction
of highly entangled Wasserstein neurons, and model performance suffers more
when these are sparsified as opposed to others.

我们展示了如何通过将 LLM 扩展为稀疏专家的混合体来提高其推理效率，其中每个专家是原始权重的副本，经过一次性修剪以特定输入值簇的方式修剪。我们称这种方法为 ' 稀疏扩展 '。我们展示了对于像 LLama 270B 这样的模型，随着稀疏专家的数量增加，稀疏扩展在相同推理 FLOP 预算下胜过所有其他一次性稀疏化方法，并且随着稀疏性的增加，这种差距加大，导致推理加速。

稀疏展开和神经元解缠

Sparse Expansion and Neuronal Disentanglement

We consider linear inverse problems where the solution is assumed to have a
sparse expansion on an arbitrary pre-assigned orthonormal basis. We prove that
replacing the usual quadratic regularizing penalties by weighted l^p-penalties
on the coefficients of such expansions, with 1 < or = p < or =2, still
regularizes the problem. If p < 2, regularized solutions of such l^p-penalized
problems will have sparser expansions, with respect to the basis under
consideration. To compute the corresponding regularized solutions we propose an
iterative algorithm that amounts to a Landweber iteration with thresholding (or
nonlinear shrinkage) applied at each iteration step. We prove that this
algorithm converges in norm. We also review some potential applications of this
method.

该研究考虑线性反问题，其中解被假定在任意预先分配的正交基上具有稀疏展开，利用加权 l^p 准则对系数进行正则化，提出了一种迭代算法计算对应的正则化解，并证明了算法的收敛性以及该方法的一些潜在应用。