Adversarial training (AT) is an important and attractive topic in deep learning security, exhibiting mysteries and odd properties. Recent studies of neural network training dynamics based on Neural Tangent Kernel (NTK) make it possible to reacquaint AT and deeply analyze its properties. In this paper, we perform an in-depth investigation of AT process and properties with NTK, such as NTK evolution. We uncover three new findings that are missed in previous works. First, we disclose the impact of data normalization on AT and the importance of unbiased estimators in batch normalization layers. Second, we experimentally explore the kernel dynamics and propose more time-saving AT methods. Third, we study the spectrum feature inside the kernel to address the catastrophic overfitting problem. To the best of our knowledge, it is the first work leveraging the observations of kernel dynamics to improve existing AT methods.

本文使用神经切线核（NTK）对敌对训练（AT）过程和性质进行了深入研究，揭示了数据归一化对AT的影响以及批归一化层中无偏估计器的重要性，并通过实验探索了内核动力学和提出了更节省时间的AT方法，同时利用内核内的频谱特征解决了灾难性过拟合问题。据我们所知，这是首个利用内核动力学观察改进现有AT方法的研究。

用神经切线核重新思考对抗训练