The mean of an unknown variance-$\sigma^2$ distribution $f$ can be estimated
from $n$ samples with variance $\frac{\sigma^2}{n}$ and nearly corresponding
subgaussian rate. When $f$ is known up to translation, this can be improved
asymptotically to $\frac{1}{n\mathcal I}$, where $\mathcal I$ is the Fisher
information of the distribution. Such an improvement is not possible for
general unknown $f$, but [Stone, 1975] showed that this asymptotic convergence
$\textit{is}$ possible if $f$ is $\textit{symmetric}$ about its mean. Stone's
bound is asymptotic, however: the $n$ required for convergence depends in an
unspecified way on the distribution $f$ and failure probability $\delta$. In
this paper we give finite-sample guarantees for symmetric mean estimation in
terms of Fisher information. For every $f, n, \delta$ with $n > \log
\frac{1}{\delta}$, we get convergence close to a subgaussian with variance
$\frac{1}{n \mathcal I_r}$, where $\mathcal I_r$ is the $r$-$\textit{smoothed}$
Fisher information with smoothing radius $r$ that decays polynomially in $n$.
Such a bound essentially matches the finite-sample guarantees in the known-$f$
setting.

本文介绍了使用 Fisher 信息进行对称分布均值估计的一种方法，并给出了收敛到亚高斯分布的有限样本保证。

具有费舍尔信息速率的有限样本对称均值估计

Finite-Sample Symmetric Mean Estimation with Fisher Information Rate

We identify and study two common failure modes for early training in deep
ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid
it, for fully connected and residual architectures. The first failure mode,
exploding/vanishing mean activation length, can be avoided by initializing
weights from a symmetric distribution with variance 2/fan-in and, for ResNets,
by correctly weighting the residual modules. We prove that the second failure
mode, exponentially large variance of activation length, never occurs in
residual nets once the first failure mode is avoided. In contrast, for fully
connected nets, we prove that this failure mode can happen and is avoided by
keeping constant the sum of the reciprocals of layer widths. We demonstrate
empirically the effectiveness of our theoretical results in predicting when
networks are able to start training. In particular, we note that many popular
initializations fail our criteria, whereas correct initialization and
architecture allows much deeper networks to be trained.

本文针对深度 ReLU 神经网络早期训练的两种常见失效模式进行研究和识别，为每种模式提供了严格的证明以及如何避免。我们证明了正确的权重初始化以及架构可以使得深层神经网络成功训练，并在实验中证实了我们的理论结果的有效性。