This article provides a comprehensive understanding of optimization in deep
learning, with a primary focus on the challenges of gradient vanishing and
gradient exploding, which normally lead to diminished model representational
ability and training instability, respectively. We analyze these two challenges
through several strategic measures, including the improvement of gradient flow
and the imposition of constraints on a network's Lipschitz constant. To help
understand the current optimization methodologies, we categorize them into two
classes: explicit optimization and implicit optimization. Explicit optimization
methods involve direct manipulation of optimizer parameters, including weight,
gradient, learning rate, and weight decay. Implicit optimization methods, by
contrast, focus on improving the overall landscape of a network by enhancing
its modules, such as residual shortcuts, normalization methods, attention
mechanisms, and activations. In this article, we provide an in-depth analysis
of these two optimization classes and undertake a thorough examination of the
Jacobian matrices and the Lipschitz constants of many widely used deep learning
modules, highlighting existing issues as well as potential improvements.
Moreover, we also conduct a series of analytical experiments to substantiate
our theoretical discussions. This article does not aim to propose a new
optimizer or network. Rather, our intention is to present a comprehensive
understanding of optimization in deep learning. We hope that this article will
assist readers in gaining a deeper insight in this field and encourages the
development of more robust, efficient, and high-performing models.

这篇论文全面介绍了深度学习中的优化，重点关注梯度消失和梯度爆炸带来的挑战，提出了改进梯度流和对网络的 Lipschitz 常数施加限制等策略，并将优化方法分为两类：显式优化和隐式优化。通过对多个深度学习模块的雅可比矩阵和 Lipschitz 常数进行分析和实验，为读者提供了深入的理解。

深度学习优化理解

Understanding Optimization of Deep Learning

Deep ResNet architectures have achieved state of the art performance on many
tasks. While they solve the problem of gradient vanishing, they might suffer
from gradient exploding as the depth becomes large (Yang et al. 2017).
Moreover, recent results have shown that ResNet might lose expressivity as the
depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these
issues, we introduce a new class of ResNet architectures, called Stable ResNet,
that have the property of stabilizing the gradient while ensuring expressivity
in the infinite depth limit.

介绍了一种新的 ResNet 架构，称为 Stable ResNet，解决了在深度增加时梯度消失和梯度爆炸的问题，并确保在无限深度时保持表达能力。

稳定的 ResNet

Stable ResNet

In order to alleviate the notorious mode collapse phenomenon in generative
adversarial networks (GANs), we propose a novel training method of GANs in
which certain fake samples are considered as real ones during the training
process. This strategy can reduce the gradient value that generator receives in
the region where gradient exploding happens. We show the process of an
unbalanced generation and a vicious circle issue resulted from gradient
exploding in practical training, which explains the instability of GANs. We
also theoretically prove that gradient exploding can be alleviated by
penalizing the difference between discriminator outputs and fake-as-real
consideration for very close real and fake samples. Accordingly, Fake-As-Real
GAN (FARGAN) is proposed with a more stable training process and a more
faithful generated distribution. Experiments on different datasets verify our
theoretical analysis.

提出了一种新的生成对抗网络训练方法 Fake-As-Real GAN（FARGAN），该方法通过将某些虚假样本在训练过程中视为真实样本，从而减少生成器在梯度爆炸区域接收的梯度值，理论上证明了梯度爆炸可以通过对真假样本的区分来减轻，实验证明该方法可以有效地解决 GAN 中的模式崩溃问题，并提高生成样本质量。