The softmax gating function is arguably the most popular choice in mixture of
experts modeling. Despite its widespread use in practice, softmax gating may
lead to unnecessary competition among experts, potentially causing the
undesirable phenomenon of representation collapse due to its inherent
structure. In response, the sigmoid gating function has been recently proposed
as an alternative and has been demonstrated empirically to achieve superior
performance. However, a rigorous examination of the sigmoid gating function is
lacking in current literature. In this paper, we verify theoretically that
sigmoid gating, in fact, enjoys a higher sample efficiency than softmax gating
for the statistical task of expert estimation. Towards that goal, we consider a
regression framework in which the unknown regression function is modeled as a
mixture of experts, and study the rates of convergence of the least squares
estimator in the over-specified case in which the number of experts fitted is
larger than the true value. We show that two gating regimes naturally arise
and, in each of them, we formulate identifiability conditions for the expert
functions and derive the corresponding convergence rates. In both cases, we
find that experts formulated as feed-forward networks with commonly used
activation such as $\mathrm{ReLU}$ and $\mathrm{GELU}$ enjoy faster convergence
rates under sigmoid gating than softmax gating. Furthermore, given the same
choice of experts, we demonstrate that the sigmoid gating function requires a
smaller sample size than its softmax counterpart to attain the same error of
expert estimation and, therefore, is more sample efficient.

通过理论验证，本文发现 sigmoid 门函数对于专家估计统计任务的样本效率比 softmax 门函数更高，而且在两个不同的门控制度下使用前馈网络作为专家模型的收敛速度更快。