The training process of ReLU neural networks often exhibits complicated
nonlinear phenomena. The nonlinearity of models and non-convexity of loss pose
significant challenges for theoretical analysis. Therefore, most previous
theoretical works on the optimization dynamics of neural networks focus either
on local analysis (like the end of training) or approximate linear models (like
Neural Tangent Kernel). In this work, we conduct a complete theoretical
characterization of the training process of a two-layer ReLU network trained by
Gradient Flow on a linearly separable data. In this specific setting, our
analysis captures the whole optimization process starting from random
initialization to final convergence. Despite the relatively simple model and
data that we studied, we reveal four different phases from the whole training
process showing a general simplifying-to-complicating learning trend. Specific
nonlinear behaviors can also be precisely identified and captured
theoretically, such as initial condensation, saddle-to-plateau dynamics,
plateau escape, changes of activation patterns, learning with increasing
complexity, etc.

本研究通过对 ReLU 神经网络的训练过程进行理论性分析，揭示了从随机初始化到最终收敛的整个优化过程，并发现了四个不同阶段，显示了一个从简单到复杂的总体趋势，此外还可以精确地识别和捕捉特定的非线性行为，如初始凝结、鞍点到高原动态、平台逃逸、激活模式的变化、学习随着复杂度的增加等现象。

理解 ReLU 网络的多阶段优化动态和丰富非线性行为

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear  Behaviors of ReLU Networks

High-dimensional partial differential equations (PDE) appear in a number of
models from the financial industry, such as in derivative pricing models,
credit valuation adjustment (CVA) models, or portfolio optimization models. The
PDEs in such applications are high-dimensional as the dimension corresponds to
the number of financial assets in a portfolio. Moreover, such PDEs are often
fully nonlinear due to the need to incorporate certain nonlinear phenomena in
the model such as default risks, transaction costs, volatility uncertainty
(Knightian uncertainty), or trading constraints in the model. Such
high-dimensional fully nonlinear PDEs are exceedingly difficult to solve as the
computational effort for standard approximation methods grows exponentially
with the dimension. In this work we propose a new method for solving
high-dimensional fully nonlinear second-order PDEs. Our method can in
particular be used to sample from high-dimensional nonlinear expectations. The
method is based on (i) a connection between fully nonlinear second-order PDEs
and second-order backward stochastic differential equations (2BSDEs), (ii) a
merged formulation of the PDE and the 2BSDE problem, (iii) a temporal forward
discretization of the 2BSDE and a spatial approximation via deep neural nets,
and (iv) a stochastic gradient descent-type optimization procedure. Numerical
results obtained using ${\rm T{\small ENSOR}F{\small LOW}}$ in ${\rm P{\small
YTHON}}$ illustrate the efficiency and the accuracy of the method in the cases
of a $100$-dimensional Black-Scholes-Barenblatt equation, a $100$-dimensional
Hamilton-Jacobi-Bellman equation, and a nonlinear expectation of a $ 100
$-dimensional $ G $-Brownian motion.

本文介绍一种新的用于解决高维金融模型中的非线性偏微分方程的方法，该方法包含非线性现象、深度神经网络和随机梯度下降类型优化过程，并通过海量数据的数值结果证明了该方法的高效性和精确性。

高维完全非线性偏微分方程和二阶反向随机微分方程的机器学习逼近算法

Machine learning approximation algorithms for high-dimensional fully  nonlinear partial differential equations and second-order backward stochastic  differential equations

Despite the widespread practical success of deep learning methods, our
theoretical understanding of the dynamics of learning in deep neural networks
remains quite sparse. We attempt to bridge the gap between the theory and
practice of deep learning by systematically analyzing learning dynamics for the
restricted case of deep linear neural networks. Despite the linearity of their
input-output map, such networks have nonlinear gradient descent dynamics on
weights that change with the addition of each new hidden layer. We show that
deep linear networks exhibit nonlinear learning phenomena similar to those seen
in simulations of nonlinear networks, including long plateaus followed by rapid
transitions to lower error solutions, and faster convergence from greedy
unsupervised pretraining initial conditions than from random initial
conditions. We provide an analytical description of these phenomena by finding
new exact solutions to the nonlinear dynamics of deep learning. Our theoretical
analysis also reveals the surprising finding that as the depth of a network
approaches infinity, learning speed can nevertheless remain finite: for a
special class of initial conditions on the weights, very deep networks incur
only a finite, depth independent, delay in learning speed relative to shallow
networks. We show that, under certain conditions on the training data,
unsupervised pretraining can find this special class of initial conditions,
while scaled random Gaussian initializations cannot. We further exhibit a new
class of random orthogonal initial conditions on weights that, like
unsupervised pre-training, enjoys depth independent learning times. We further
show that these initial conditions also lead to faithful propagation of
gradients even in deep nonlinear networks, as long as they operate in a special
regime known as the edge of chaos.

通过对深度线性神经网络的学习动态进行系统分析，我们发现这些网络表现出类似于非线性神经网络的非线性学习现象，包括长时间的平原，然后快速转换到更低误差的解决方案，以及从贪婪的无监督预训练初始条件下的更快收敛等。同时，我们发现在权重的某些特殊初始条件下，非监督预训练可以找到这些初始条件，同时表现出深度独立的学习时间，而随机高斯初始化则做不到。