Data augmentation (DA) techniques aim to increase data variability, and thus train deep networks with better generalisation. The pioneering AutoAugment automated the search for optimal DA policies with reinforcement learning. However, AutoAugment is extremely computationally expensive, limiting its wide applicability. Followup work such as PBA and Fast AutoAugment improved efficiency, but optimization speed remains a bottleneck. In this paper, we propose Differentiable Automatic Data Augmentation (DADA) which dramatically reduces the cost. DADA relaxes the discrete DA policy selection to a differentiable optimization problem via Gumbel-Softmax. In addition, we introduce an unbiased gradient estimator, RELAX, leading to an efficient and effective one-pass optimization strategy to learn an efficient and accurate DA policy. We conduct extensive experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. Furthermore, we demonstrate the value of Auto DA in pre-training for downstream detection problems. Results show our DADA is at least one order of magnitude faster than the state-of-the-art while achieving very comparable accuracy.

本研究提出Differentiable Automatic Data Augmentation (DADA)算法，使用Gumbel-Softmax方法实现对离散的数据增强方法的选择，引入RELAX无偏梯度估计方法，提高了数据增强策略的学习效率。在CIFAR-10、CIFAR-100、SVHN和ImageNet数据集上的实验证明了该算法在达到同等精度的情况下，比现有最先进算法至少快一个数量级。同时，研究还证明了自动数据增强在预训练方面的价值。

DADA:可微分自动数据增强