Recent research in robust optimization has shown an overfitting-like
phenomenon in which models trained against adversarial attacks exhibit higher
robustness on the training set compared to the test set. Although previous work
provided theoretical explanations for this phenomenon using a robust
PAC-Bayesian bound over the adversarial test error, related algorithmic
derivations are at best only loosely connected to this bound, which implies
that there is still a gap between their empirical success and our understanding
of adversarial robustness theory. To close this gap, in this paper we consider
a different form of the robust PAC-Bayesian bound and directly minimize it with
respect to the model posterior. The derivation of the optimal solution connects
PAC-Bayesian learning to the geometry of the robust loss surface through a
Trace of Hessian (TrH) regularizer that measures the surface flatness. In
practice, we restrict the TrH regularizer to the top layer only, which results
in an analytical solution to the bound whose computational cost does not depend
on the network depth. Finally, we evaluate our TrH regularization approach over
CIFAR-10/100 and ImageNet using Vision Transformers (ViT) and compare against
baseline adversarial robustness algorithms. Experimental results show that TrH
regularization leads to improved ViT robustness that either matches or
surpasses previous state-of-the-art approaches while at the same time requires
less memory and computational cost.

本文研究了鲁棒优化中的过拟合现象，提出了一种不同形式的鲁棒 PAC-Bayesian 边界并通过迹（TrH）正则化器将 PAC-Bayesian 学习与鲁棒损失面的几何形状连接，实验结果证明了该方法能够提高视觉变换器的鲁棒性且计算成本低。