Top-K sparse softmax gating mixture of experts has been widely used for
scaling up massive deep-learning architectures without increasing the
computational cost. Despite its popularity in real-world applications, the
theoretical understanding of that gating function has remained an open problem.
The main challenge comes from the structure of the top-K sparse softmax gating
function, which partitions the input space into multiple regions with distinct
behaviors. By focusing on a Gaussian mixture of experts, we establish
theoretical results on the effects of the top-K sparse softmax gating function
on both density and parameter estimations. Our results hinge upon defining
novel loss functions among parameters to capture different behaviors of the
input regions. When the true number of experts $k_{\ast}$ is known, we
demonstrate that the convergence rates of density and parameter estimations are
both parametric on the sample size. However, when $k_{\ast}$ becomes unknown
and the true model is over-specified by a Gaussian mixture of $k$ experts where
$k > k_{\ast}$, our findings suggest that the number of experts selected from
the top-K sparse softmax gating function must exceed the total cardinality of a
certain number of Voronoi cells associated with the true parameters to
guarantee the convergence of the density estimation. Moreover, while the
density estimation rate remains parametric under this setting, the parameter
estimation rates become substantially slow due to an intrinsic interaction
between the softmax gating and expert functions.

通过定义损失函数和构建高斯混合专家模型，我们从密度和参数估计的角度对顶部 K 个稀疏 Softmax 门控函数进行理论研究，发现要保证密度估计的收敛，所选取的门控函数所包含的专家数量至少必须超过与真实参数相关的某些 Voronoi 单元总基数，并且由于 Softmax 门控和专家函数之间的内在相互作用，参数估计速度明显变慢。