We study the problem of alleviating the instability issue in the GAN training
procedure via new architecture design. The discrepancy between the minimax and
maximin objective values could serve as a proxy for the difficulties that the
alternating gradient descent encounters in the optimization of GANs. In this
work, we give new results on the benefits of multi-generator architecture of
GANs. We show that the minimax gap shrinks to $\epsilon$ as the number of
generators increases with rate $\widetilde{O}(1/\epsilon)$. This improves over
the best-known result of $\widetilde{O}(1/\epsilon^2)$. At the core of our
techniques is a novel application of Shapley-Folkman lemma to the generic
minimax problem, where in the literature the technique was only known to work
when the objective function is restricted to the Lagrangian function of a
constraint optimization problem. Our proposed Stackelberg GAN performs well
experimentally in both synthetic and real-world datasets, improving Fr\'echet
Inception Distance by $14.61\%$ over the previous multi-generator GANs on the
benchmark datasets.

通过新的架构设计，我们通过多种生成器体系结构缓解了 GAN 训练中的不稳定性问题，并通过 Shapley-Folkman 引理的新应用提出了一种新的 Stackelberg GAN，优于以前的多生成器 GAN，在基准数据集上比以前的方法提高了 14.61% 的 Fréchet Inception 距离。

Stackelberg GAN：通过多生成器结构实现可证明的极小极大均衡

Stackelberg GAN: Towards Provable Minimax Equilibrium via  Multi-Generator Architectures

We give a rigorous analysis of the statistical behavior of gradients in a
randomly initialized fully connected network N with ReLU activations. Our
results show that the empirical variance of the squares of the entries in the
input-output Jacobian of N is exponential in a simple architecture-dependent
constant beta, given by the sum of the reciprocals of the hidden layer widths.
When beta is large, the gradients computed by N at initialization vary wildly.
Our approach complements the mean field theory analysis of random networks.
From this point of view, we rigorously compute finite width corrections to the
statistics of gradients at the edge of chaos.

我们对一个使用 ReLU 激活的随机初始化全连接网络 N 的梯度统计行为进行了严格分析，发现其输出输入 Jacobian 的平方条目的经验方差呈现简单依赖架构的常数 beta 的指数形式，该常数由隐藏层宽度的倒数之和组成，当 beta 很大时，N 在初始化时计算的梯度变化异常剧烈。我们的方法补充了随机网络的平均场理论分析，从这个视角出发，我们对处于混沌边缘的梯度统计的有限宽度修正进行了严格计算。

哪些神经网络架构会导致梯度爆炸和梯度消失？

Which Neural Net Architectures Give Rise To Exploding and Vanishing  Gradients?

We study characteristics of receptive fields of units in deep convolutional
networks. The receptive field size is a crucial issue in many visual tasks, as
the output must respond to large enough areas in the image to capture
information about large objects. We introduce the notion of an effective
receptive field, and show that it both has a Gaussian distribution and only
occupies a fraction of the full theoretical receptive field. We analyze the
effective receptive field in several architecture designs, and the effect of
nonlinear activations, dropout, sub-sampling and skip connections on it. This
leads to suggestions for ways to address its tendency to be too small.

研究了深度卷积网络单元感受野的特性，提出了有效感受野的概念，并分析了它在几种架构设计中的作用，以及非线性激活、dropout、下采样和跳跃连接对其产生的影响。结果表明，这些因素对有效感受野的大小具有影响，并提出了一些解决途径。