Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone
approach in imitation learning. This paper investigates the gradient explosion
in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with
stochastic policy (ST-GAIL). We begin with the observation that the training
can be highly unstable for DE-GAIL at the beginning of the training phase and
end up divergence. Conversely, the ST-GAIL training trajectory remains
consistent, reliably converging. To shed light on these disparities, we provide
an explanation from a theoretical perspective. By establishing a probabilistic
lower bound for GAIL, we demonstrate that gradient explosion is an inevitable
outcome for DE-GAIL due to occasionally large expert-imitator policy disparity,
whereas ST-GAIL does not have the issue with it. To substantiate our assertion,
we illustrate how modifications in the reward function can mitigate the
gradient explosion challenge. Finally, we propose CREDO, a simple yet effective
strategy that clips the reward function during the training phase, allowing the
GAIL to enjoy high data efficiency and stable trainability.

通过对 DE-GAIL 和 ST-GAIL 的研究，本文从理论角度解释了 gradient explosion 在 DE-GAIL 中是不可避免的问题，并提出了使用 CREDO 策略通过限制奖励函数来解决 gradient explosion 挑战，从而使 GAIL 在训练过程中获得较高的数据效率和稳定性。

生成对抗性模仿学习中的梯度爆炸探索：概率视角

Exploring Gradient Explosion in Generative Adversarial Imitation  Learning: A Probabilistic Perspective

Deep neural networks based on batch normalization and ReLU-like activation
functions can experience instability during the early stages of training due to
the high gradient induced by temporal gradient explosion. We explain how ReLU
reduces variance more than expected, and how batch normalization amplifies the
gradient during recovery, which causes gradient explosion while forward
propagation remains stable. Additionally, we discuss how the dynamics of a deep
neural network change during training and how the correlation between inputs
can alleviate this problem. Lastly, we propose a better adaptive learning rate
algorithm inspired by second-order optimization algorithms, which outperforms
existing learning rate scaling methods in large batch training and can also
replace WarmUp in small batch training.

讲述了基于批归一化和 ReLU 的深度神经网络存在于训练早期不稳定的情况，提出了更好的适应性学习率算法来替代现有的学习率缩放方法，该算法在大批量训练中表现优于现有方法。

BN 与 ReLU 之间的不协调导致梯度爆炸，但被激活之间的相关性所抵消

The Disharmony Between BN and ReLU Causes Gradient Explosion, but is  Offset by the Correlation Between Activations

Stochastic convex optimization is a basic and well studied primitive in
machine learning. It is well known that convex and Lipschitz functions can be
minimized efficiently using Stochastic Gradient Descent (SGD). The Normalized
Gradient Descent (NGD) algorithm, is an adaptation of Gradient Descent, which
updates according to the direction of the gradients, rather than the gradients
themselves. In this paper we analyze a stochastic version of NGD and prove its
convergence to a global minimum for a wider class of functions: we require the
functions to be quasi-convex and locally-Lipschitz. Quasi-convexity broadens
the con- cept of unimodality to multidimensions and allows for certain types of
saddle points, which are a known hurdle for first-order optimization methods
such as gradient descent. Locally-Lipschitz functions are only required to be
Lipschitz in a small region around the optimum. This assumption circumvents
gradient explosion, which is another known hurdle for gradient descent
variants. Interestingly, unlike the vanilla SGD algorithm, the stochastic
normalized gradient descent algorithm provably requires a minimal minibatch
size.

本文研究随机版归一化梯度下降算法，并证明了该算法在优化拥有拟凸和局部 Lipschitz 性质的函数时，能够保证收敛到全局最优解。与标准的随机梯度下降算法不同的是，该算法要求使用最小的小批量大小，以避免梯度爆炸等问题。