Data Augmentation (DA) is known to improve the generalizability of deep
neural networks. Most existing DA techniques naively add a certain number of
augmented samples without considering the quality and the added computational
cost of these samples. To tackle this problem, a common strategy, adopted by
several state-of-the-art DA methods, is to adaptively generate or re-weight
augmented samples with respect to the task objective during training. However,
these adaptive DA methods: (1) are computationally expensive and not
sample-efficient, and (2) are designed merely for a specific setting. In this
work, we present a universal DA technique, called Glitter, to overcome both
issues. Glitter can be plugged into any DA method, making training
sample-efficient without sacrificing performance. From a pre-generated pool of
augmented samples, Glitter adaptively selects a subset of worst-case samples
with maximal loss, analogous to adversarial DA. Without altering the training
strategy, the task objective can be optimized on the selected subset. Our
thorough experiments on the GLUE benchmark, SQuAD, and HellaSwag in three
widely used training setups including consistency training, self-distillation
and knowledge distillation reveal that Glitter is substantially faster to train
and achieves a competitive performance, compared to strong baselines.

本文提出了一种名为 Glitter 的通用数据增强技术，该技术可以自适应地选择最麻烦的样本并通过优化它们来提高样本效率，从而使深度神经网络在训练过程中不会牺牲性能。