Motivated by classical work on the numerical integration of ordinary
differential equations we present a ResNet-styled neural network architecture
that encodes non-expansive (1-Lipschitz) operators, as long as the spectral
norms of the weights are appropriately constrained. This is to be contrasted
with the ordinary ResNet architecture which, even if the spectral norms of the
weights are constrained, has a Lipschitz constant that, in the worst case,
grows exponentially with the depth of the network. Further analysis of the
proposed architecture shows that the spectral norms of the weights can be
further constrained to ensure that the network is an averaged operator, making
it a natural candidate for a learned denoiser in Plug-and-Play algorithms.
Using a novel adaptive way of enforcing the spectral norm constraints, we show
that, even with these constraints, it is possible to train performant networks.
The proposed architecture is applied to the problem of adversarially robust
image classification, to image denoising, and finally to the inverse problem of
deblurring.

本文提出了一种 ResNet 风格的神经网络架构，编码非扩张（1-Lipschitz）算子，不同于普通的 ResNet 架构，该架构的 Lipschitz 常数不会随着网络深度的增加而呈指数级增长。进一步分析表明，权重的谱范数可以进一步约束，以确保网络是平均算子，使其成为 Plug-and-Play 算法中学习去噪器的自然候选物。通过一种新颖的自适应方法实现了谱范数约束，证明了即使有这些约束，也可以训练出性能良好的网络。提出的架构应用于对抗鲁棒图像分类问题，图像去噪以及反问题退化模糊。

利用凸分析和 ODE 设计稳定的神经网络

Designing Stable Neural Networks using Convex Analysis and ODEs

Recurrent Neural Networks (RNNs) have been widely applied to sequential data
analysis. Due to their complicated modeling structures, however, the theory
behind is still largely missing. To connect theory and practice, we study the
generalization properties of vanilla RNNs as well as their variants, including
Minimal Gated Unit (MGU), Long Short Term Memory (LSTM), and Convolutional
(Conv) RNNs. Specifically, our theory is established under the PAC-Learning
framework. The generalization bound is presented in terms of the spectral norms
of the weight matrices and the total number of parameters. We also establish
refined generalization bounds with additional norm assumptions, and draw a
comparison among these bounds. We remark: (1) Our generalization bound for
vanilla RNNs is significantly tighter than the best of existing results; (2) We
are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs
in the exiting literature; (3) We demonstrate the advantages of these variants
in generalization.

研究对循环神经网络的推广能力，通过探究四种变体，包括最小门单元、长短期记忆和卷积循环神经网络，建立了 PAC-Learning 框架下的推广界限，并在额外的规范假设下建立了更加精细的推广界限，与现有结果相比，说明了这些变体在推广中的优势。

一类循环神经网络的泛化界限

On Generalization Bounds of a Family of Recurrent Neural Networks

The ability of overparameterized deep networks to generalize well has been
linked to the fact that stochastic gradient descent (SGD) finds solutions that
lie in flat, wide minima in the training loss -- minima where the output of the
network is resilient to small random noise added to its parameters. So far this
observation has been used to provide generalization guarantees only for neural
networks whose parameters are either \textit{stochastic} or
\textit{compressed}. In this work, we present a general PAC-Bayesian framework
that leverages this observation to provide a bound on the original network
learned -- a network that is deterministic and uncompressed. What enables us to
do this is a key novelty in our approach: our framework allows us to show that
if on training data, the interactions between the weight matrices satisfy
certain conditions that imply a wide training loss minimum, these conditions
themselves {\em generalize} to the interactions between the matrices on test
data, thereby implying a wide test loss minimum. We then apply our general
framework in a setup where we assume that the pre-activation values of the
network are not too small (although we assume this only on the training data).
In this setup, we provide a generalization guarantee for the original
(deterministic, uncompressed) network, that does not scale with product of the
spectral norms of the weight matrices -- a guarantee that would not have been
possible with prior approaches.

本文研究了过参数化的深层网络使用随机梯度下降法（SGD）能够良好推广的能力，提出了一种 PAC-Bayesian 框架，利用这种能力为原始网络提供界限，同时不会受到权重矩阵谱范数乘积的影响。