Recent studies on neural networks with pre-trained weights (i.e., BERT) have
mainly focused on a low-dimensional subspace, where the embedding vectors
computed from input words (or their contexts) are located. In this work, we
propose a new approach to finding and regularizing the remainder of the space,
referred to as out-of-manifold, which cannot be accessed through the words.
Specifically, we synthesize the out-of-manifold embeddings based on two
embeddings obtained from actually-observed words, to utilize them for
fine-tuning the network. A discriminator is trained to detect whether an input
embedding is located inside the manifold or not, and simultaneously, a
generator is optimized to produce new embeddings that can be easily identified
as out-of-manifold by the discriminator. These two modules successfully
collaborate in a unified and end-to-end manner for regularizing the
out-of-manifold. Our extensive evaluation on various text classification
benchmarks demonstrates the effectiveness of our approach, as well as its good
compatibility with existing data augmentation techniques which aim to enhance
the manifold.

本文提出了一种新的方法来找到和规范称为流形之外的剩余空间，利用两个从实际观察到的单词获得的嵌入，我们合成了流形之外的嵌入，训练鉴别器检测输入的嵌入是否位于流形之内，同时优化生成器。通过在各种文本分类基准上进行全面评估，证明了我们的方法的有效性以及其良好兼容性。