We study the problem of learning one-hidden-layer neural networks with
Rectified Linear Unit (ReLU) activation function, where the inputs are sampled
from standard Gaussian distribution and the outputs are generated from a noisy
teacher network. We analyze the performance of gradient descent for training
such kind of neural networks based on empirical risk minimization, and provide
algorithm-dependent guarantees. In particular, we prove that tensor
initialization followed by gradient descent can converge to the ground-truth
parameters at a linear rate up to some statistical error. To the best of our
knowledge, this is the first work characterizing the recovery guarantee for
practical learning of one-hidden-layer ReLU networks with multiple neurons.
Numerical experiments verify our theoretical findings.

本文研究从标准高斯分布采样输入，从嘈杂的教师网络生成输出的一层隐藏神经网络的学习问题。研究分析了梯度下降在基于经验风险最小化的训练中的性能，并提供了算法相关的保证，证明了张量初始化后跟随梯度下降可以以线性速率收敛到地面真值参数，证明本文是第一个表征实际学习具有多个神经元的一层 ReLU 网络的恢复保证的工作。数值实验验证了我们的理论发现。