The softmax gating function is arguably the most popular choice in mixture of
experts modeling. Despite its widespread use in practice, softmax gating may
lead to unnecessary competition among experts, potentially causing the
undesirable phenomenon of representation collapse due to its inherent
structure. In response, the sigmoid gating function has been recently proposed
as an alternative and has been demonstrated empirically to achieve superior
performance. However, a rigorous examination of the sigmoid gating function is
lacking in current literature. In this paper, we verify theoretically that
sigmoid gating, in fact, enjoys a higher sample efficiency than softmax gating
for the statistical task of expert estimation. Towards that goal, we consider a
regression framework in which the unknown regression function is modeled as a
mixture of experts, and study the rates of convergence of the least squares
estimator in the over-specified case in which the number of experts fitted is
larger than the true value. We show that two gating regimes naturally arise
and, in each of them, we formulate identifiability conditions for the expert
functions and derive the corresponding convergence rates. In both cases, we
find that experts formulated as feed-forward networks with commonly used
activation such as $\mathrm{ReLU}$ and $\mathrm{GELU}$ enjoy faster convergence
rates under sigmoid gating than softmax gating. Furthermore, given the same
choice of experts, we demonstrate that the sigmoid gating function requires a
smaller sample size than its softmax counterpart to attain the same error of
expert estimation and, therefore, is more sample efficient.

通过理论验证，本文发现 sigmoid 门函数对于专家估计统计任务的样本效率比 softmax 门函数更高，而且在两个不同的门控制度下使用前馈网络作为专家模型的收敛速度更快。

混合专家模型中，Sigmoid 门控比 Softmax 门控的样本效率更高

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture  of Experts

Understanding parameter estimation of softmax gating Gaussian mixture of
experts has remained a long-standing open problem in the literature. It is
mainly due to three fundamental theoretical challenges associated with the
softmax gating: (i) the identifiability only up to the translation of the
parameters; (ii) the intrinsic interaction via partial differential equation
between the softmax gating and the expert functions in Gaussian distribution;
(iii) the complex dependence between the numerator and denominator of the
conditional density of softmax gating Gaussian mixture of experts. We resolve
these challenges by proposing novel Vononoi loss functions among parameters and
establishing the convergence rates of the maximum likelihood estimator (MLE)
for solving parameter estimation in these models. When the number of experts is
unknown and over-specified, our findings show a connection between the rate of
MLE and a solvability problem of a system of polynomial equations.

通过引入新的 Vononoi 损失函数，建立最大似然估计器的收敛速度，解决了 softmax gating 高斯混合专家的参数估计中的三种理论挑战。