We propose local distributional smoothness (LDS), a new notion of smoothness
for statistical model that can be used as a regularization term to promote the
smoothness of the model distribution. We named the LDS based regularization as
virtual adversarial training (VAT). The LDS of a model at an input datapoint is
defined as the KL-divergence based robustness of the model distribution against
local perturbation around the datapoint. VAT resembles adversarial training,
but distinguishes itself in that it determines the adversarial direction from
the model distribution alone without using the label information, making it
applicable to semi-supervised learning. The computational cost for VAT is
relatively low. For neural network, the approximated gradient of the LDS can be
computed with no more than three pairs of forward and back propagations. When
we applied our technique to supervised and semi-supervised learning for the
MNIST dataset, it outperformed all the training methods other than the current
state of the art method, which is based on a highly advanced generative model.
We also applied our method to SVHN and NORB, and confirmed our method's
superior performance over the current state of the art semi-supervised method
applied to these datasets.

本文提出了一种基于 KL 分歧的局部分布平滑正则方法，称作虚拟对抗训练，用于半监督学习中的神经网络，通过模型分布的稳健性来决定对抗方向，较低的计算成本使得该方法在 MNIST，SVHN 和 NORB 数据集上的性能超过了其他训练方法，除一个高级的生成模型外取得当前最佳表现。