Despite strong performance in numerous applications, the fragility of deep
learning to input perturbations has raised serious questions about its use in
safety-critical domains. While adversarial training can mitigate this issue in
practice, state-of-the-art methods are increasingly application-dependent,
heuristic in nature, and suffer from fundamental trade-offs between nominal
performance and robustness. Moreover, the problem of finding worst-case
perturbations is non-convex and underparameterized, both of which engender a
non-favorable optimization landscape. Thus, there is a gap between the theory
and practice of adversarial training, particularly with respect to when and why
adversarial training works. In this paper, we take a constrained learning
approach to address these questions and to provide a theoretical foundation for
robust learning. In particular, we leverage semi-infinite optimization and
non-convex duality theory to show that adversarial training is equivalent to a
statistical problem over perturbation distributions, which we characterize
completely. Notably, we show that a myriad of previous robust training
techniques can be recovered for particular, sub-optimal choices of these
distributions. Using these insights, we then propose a hybrid Langevin Monte
Carlo approach of which several common algorithms (e.g., PGD) are special
cases. Finally, we show that our approach can mitigate the trade-off between
nominal and robust performance, yielding state-of-the-art results on MNIST and
CIFAR-10. Our code is available at: this https URL

本文通过半无限优化和非凸对偶理论的研究，证明对抗性训练等价于在扰动分布上的统计问题，并对此进行完整的表征。我们提出一种基于 Langevin Monte Carlo 的混合方法，可以缓解鲁棒性与标准性能之间的平衡问题，并取得了 MNIST 和 CIFAR-10 等领域最先进的结果。