Adversarial training is a standard method to train deep neural networks to be
robust to adversarial perturbation. Similar to surprising $\textit{clean
generalization}$ ability in the standard deep learning setting, neural networks
trained by adversarial training also generalize well for $\textit{unseen clean
data}$. However, in constrast with clean generalization, while adversarial
training method is able to achieve low $\textit{robust training error}$, there
still exists a significant $\textit{robust generalization gap}$, which promotes
us exploring what mechanism leads to both $\textit{clean generalization and
robust overfitting (CGRO)}$ during learning process. In this paper, we provide
a theoretical understanding of this CGRO phenomenon in adversarial training.
First, we propose a theoretical framework of adversarial training, where we
analyze $\textit{feature learning process}$ to explain how adversarial training
leads network learner to CGRO regime. Specifically, we prove that, under our
patch-structured dataset, the CNN model provably partially learns the true
feature but exactly memorizes the spurious features from training-adversarial
examples, which thus results in clean generalization and robust overfitting.
For more general data assumption, we then show the efficiency of CGRO
classifier from the perspective of $\textit{representation complexity}$. On the
empirical side, to verify our theoretical analysis in real-world vision
dataset, we investigate the $\textit{dynamics of loss landscape}$ during
training. Moreover, inspired by our experiments, we prove a robust
generalization bound based on $\textit{global flatness}$ of loss landscape,
which may be an independent interest.

本文针对对抗训练中的干净泛化和鲁棒过度拟合现象，提出了一种理论框架，分析了特征学习过程，证明了部分学习真实特征和完全记忆训练对抗样本中噪声特征的机制，并从损失景观动态的角度验证了我们的理论分析，提出了基于损失景观全局平坦性的鲁棒泛化界限。

对抗性训练中为什么会出现干净泛化和鲁棒过拟合

Why Clean Generalization and Robust Overfitting Both Happen in  Adversarial Training

It has been observed in practice that applying pruning-at-initialization
methods to neural networks and training the sparsified networks can not only
retain the testing performance of the original dense models, but also sometimes
even slightly boost the generalization performance. Theoretical understanding
for such experimental observations are yet to be developed. This work makes the
first attempt to study how different pruning fractions affect the model's
gradient descent dynamics and generalization. Specifically, this work considers
a classification task for overparameterized two-layer neural networks, where
the network is randomly pruned according to different rates at the
initialization. It is shown that as long as the pruning fraction is below a
certain threshold, gradient descent can drive the training loss toward zero and
the network exhibits good generalization performance. More surprisingly, the
generalization bound gets better as the pruning fraction gets larger. To
complement this positive result, this work further shows a negative result:
there exists a large pruning fraction such that while gradient descent is still
able to drive the training loss toward zero (by memorizing noise), the
generalization performance is no better than random guessing. This further
suggests that pruning can change the feature learning process, which leads to
the performance drop of the pruned neural network.

该研究可以通过不同的修剪比率来影响模型的梯度下降动态和泛化性能，但是过高的修剪比率可能会导致模型性能下降，从而导致模型无法学习到有用的特征。

训练前进行修剪可能改善泛化性能，可以证明

Pruning Before Training May Improve Generalization, Provably

Verifying the identity of a person using handwritten signatures is
challenging in the presence of skilled forgeries, where a forger has access to
a person's signature and deliberately attempt to imitate it. In offline
(static) signature verification, the dynamic information of the signature
writing process is lost, and it is difficult to design good feature extractors
that can distinguish genuine signatures and skilled forgeries. This reflects in
a relatively poor performance, with verification errors around 7% in the best
systems in the literature. To address both the difficulty of obtaining good
features, as well as improve system performance, we propose learning the
representations from signature images, in a Writer-Independent format, using
Convolutional Neural Networks. In particular, we propose a novel formulation of
the problem that includes knowledge of skilled forgeries from a subset of users
in the feature learning process, that aims to capture visual cues that
distinguish genuine signatures and forgeries regardless of the user. Extensive
experiments were conducted on four datasets: GPDS, MCYT, CEDAR and Brazilian
PUC-PR datasets. On GPDS-160, we obtained a large improvement in
state-of-the-art performance, achieving 1.72% Equal Error Rate, compared to
6.97% in the literature. We also verified that the features generalize beyond
the GPDS dataset, surpassing the state-of-the-art performance in the other
datasets, without requiring the representation to be fine-tuned to each
particular dataset.

使用卷积神经网络和特征学习过程，以写作人无关的方式对离线签名进行识别，从而提高系统性能和对技艺熟练的伪造的辨别能力。