We investigate the training dynamics of two-layer neural networks when
learning multi-index target functions. We focus on multi-pass gradient descent
(GD) that reuses the batches multiple times and show that it significantly
changes the conclusion about which functions are learnable compared to
single-pass gradient descent. In particular, multi-pass GD with finite stepsize
is found to overcome the limitations of gradient flow and single-pass GD given
by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et
al., 2023) of the target function. We show that upon re-using batches, the
network achieves in just two time steps an overlap with the target subspace
even for functions not satisfying the staircase property (Abbe et al., 2021).
We characterize the (broad) class of functions efficiently learned in finite
time. The proof of our results is based on the analysis of the Dynamical
Mean-Field Theory (DMFT). We further provide a closed-form description of the
dynamical process of the low-dimensional projections of the weights, and
numerical experiments illustrating the theory.

使用两层神经网络学习多指标目标函数时，我们研究了训练动态。我们关注多次梯度下降（GD）使用多次批次并显示它显著改变了对可学习功能的结论，与单次梯度下降相比。特别是，我们发现具有有限步长的多次 GD 能够克服目标函数的信息指数（Ben Arous 等人，2021）和跃迁指数（Abbe 等人，2023）的限制，从而与目标子空间重叠。我们表明，在重新使用批次时，即使对于不满足阶梯特性的函数（Abbe 等人，2021），网络在仅两个时间步骤内即能与目标子空间有重叠。我们对有限时间内有效学习的（广义的）函数类进行了表征。我们的结果证明基于动态均场理论（DMFT）的分析。我们进一步提供了权重的低维投影的动态过程的闭合形式描述，并通过数值实验来说明该理论。