Deep neural networks have yielded superior performance in many applications;
however, the gradient computation in a deep model with millions of instances
lead to a lengthy training process even with modern GPU/TPU hardware
acceleration. In this paper, we propose AutoAssist, a simple framework to
accelerate training of a deep neural network. Typically, as the training
procedure evolves, the amount of improvement in the current model by a
stochastic gradient update on each instance varies dynamically. In AutoAssist,
we utilize this fact and design a simple instance shrinking operation, which is
used to filter out instances with relatively low marginal improvement to the
current model; thus the computationally intensive gradient computations are
performed on informative instances as much as possible. We prove that the
proposed technique outperforms vanilla SGD with existing importance sampling
approaches for linear SVM problems, and establish an O(1/k) convergence for
strongly convex problems. In order to apply the proposed techniques to
accelerate training of deep models, we propose to jointly train a very
lightweight Assistant network in addition to the original deep network referred
to as Boss. The Assistant network is designed to gauge the importance of a
given instance with respect to the current Boss such that a shrinking operation
can be applied in the batch generator. With careful design, we train the Boss
and Assistant in a nonblocking and asynchronous fashion such that overhead is
minimal. We demonstrate that AutoAssist reduces the number of epochs by 40% for
training a ResNet to reach the same test accuracy on an image classification
data set and saves 30% training time needed for a transformer model to yield
the same BLEU scores on a translation dataset.

本文提出 AutoAssist 框架，使用实例减少操作来加速训练深度神经网络，并通过设计一个轻量级的 Assistant 网络对 Boss 网络进行辅助来降低训练时间，实验结果表明，AutoAssist 在图像分类和翻译数据集上表现出色。