Transformers have rapidly increased in popularity in recent years, achieving state-of-the-art performance in processing text, images, audio and video. However, Transformers present large computational requirements for both training and inference, and are prone to overfitting during training. To address these challenges, we present Input Compression with Positional Consistency (ICPC), a new data augmentation method that, unlike prior augmentation techniques, simultaneously improves both generalization and training efficiency. ICPC applies varying levels of compression to each training sample in each epoch. This leads to smaller input sequences being processed by the Transformer, and hence faster training, while also alleviating overfitting by presenting each input with different compression levels. We introduce a consistency-aware position selection method in ICPC that enables accurate processing of compressed inputs without any changes to the underlying Transformer architecture. We detail compression-based augmentation methods for four different modalities -- insignificant word pruning for text, resolution modulation for images, spatio-temporal resolution modulation for videos, and spectogram size modulation for audio. ICPC also enables efficient variable-effort inference, where samples are first inferred at high compression levels, and progressively re-evaluated with lower compression for more challenging inputs. On 9 diverse tasks spanning 4 different modalities, ICPC improves accuracy by up to 1%, while also accelerating training and inference by up to 2.9X and 2.6X, respectively. Code is available at https://github.com/amrnag/ICPC.

使用输入压缩和位置一致性（ICPC）的新数据增强方法，对Transformer进行了改进，以提高泛化能力和训练效率，并减轻过拟合。ICPC通过在每个时期对每个训练样本应用不同级别的压缩，实现了更快的训练，同时通过提供不同的压缩级别来减轻过拟合。在四个不同的模态下，ICPC提供了基于压缩的数据增强方法，并在九个不同任务中提高了1%的准确性，同时加速了训练和推断。

用于Transformer神经网络高效训练和推理的位置一致性输入压缩