Dense-to-sparse gating mixture of experts (MoE) has recently become an
effective alternative to a well-known sparse MoE. Rather than fixing the number
of activated experts as in the latter model, which could limit the
investigation of potential experts, the former model utilizes the temperature
to control the softmax weight distribution and the sparsity of the MoE during
training in order to stabilize the expert specialization. Nevertheless, while
there are previous attempts to theoretically comprehend the sparse MoE, a
comprehensive analysis of the dense-to-sparse gating MoE has remained elusive.
Therefore, we aim to explore the impacts of the dense-to-sparse gate on the
maximum likelihood estimation under the Gaussian MoE in this paper. We
demonstrate that due to interactions between the temperature and other model
parameters via some partial differential equations, the convergence rates of
parameter estimations are slower than any polynomial rates, and could be as
slow as $\mathcal{O}(1/\log(n))$, where $n$ denotes the sample size. To address
this issue, we propose using a novel activation dense-to-sparse gate, which
routes the output of a linear layer to an activation function before delivering
them to the softmax function. By imposing linearly independence conditions on
the activation function and its derivatives, we show that the parameter
estimation rates are significantly improved to polynomial rates.

稠密到稀疏门控专家混合模型（MoE）是稀疏 MoE 的有效替代，本文分析了稠密到稀疏门控 MoE 对高斯 MoE 下的极大似然估计的影响，并提出了一种新的激活稠密到稀疏门，以改善参数估计速率。