The cosine router in sparse Mixture of Experts (MoE) has recently emerged as
an attractive alternative to the conventional linear router. Indeed, the cosine
router demonstrates favorable performance in image and language tasks and
exhibits better ability to mitigate the representation collapse issue, which
often leads to parameter redundancy and limited representation potentials.
Despite its empirical success, a comprehensive analysis of the cosine router in
sparse MoE has been lacking. Considering the least square estimation of the
cosine routing sparse MoE, we demonstrate that due to the intrinsic interaction
of the model parameters in the cosine router via some partial differential
equations, regardless of the structures of the experts, the estimation rates of
experts and model parameters can be as slow as $\mathcal{O}(1/\log^{\tau}(n))$
where $\tau > 0$ is some constant and $n$ is the sample size. Surprisingly,
these pessimistic non-polynomial convergence rates can be circumvented by the
widely used technique in practice to stabilize the cosine router -- simply
adding noises to the $\mathbb{L}_{2}$ norms in the cosine router, which we
refer to as \textit{perturbed cosine router}. Under the strongly identifiable
settings of the expert functions, we prove that the estimation rates for both
the experts and model parameters under the perturbed cosine routing sparse MoE
are significantly improved to polynomial rates. Finally, we conduct extensive
simulation studies in both synthetic and real data settings to empirically
validate our theoretical results.

通过对稀疏专家混合模型中余弦路由器的计算进行全面分析，我们证明当通过添加噪声到余弦路由器中的 L2 范数来稳定余弦路由器时，无论专家的结构如何，在稀疏混合模型中估计的速度可以显著提高到多项式速度。