Deep neural networks (DNN) have achieved remarkable success in various
fields, including computer vision and natural language processing. However,
training an effective DNN model still poses challenges. This paper aims to
propose a method to optimize the training effectiveness of DNN, with the goal
of improving model performance. Firstly, based on the observation that the DNN
parameters change in certain laws during training process, the potential of
parameter prediction for improving model training efficiency and performance is
discovered. Secondly, considering the magnitude of DNN model parameters,
hardware limitations and characteristics of Stochastic Gradient Descent (SGD)
for noise tolerance, a Parameter Linear Prediction (PLP) method is exploit to
perform DNN parameter prediction. Finally, validations are carried out on some
representative backbones. Experiment results show that compare to the normal
training ways, under the same training conditions and epochs, by employing
proposed PLP method, the optimal model is able to obtain average about 1%
accuracy improvement and 0.01 top-1/top-5 error reduction for Vgg16, Resnet18
and GoogLeNet based on CIFAR-100 dataset, which shown the effectiveness of the
proposed method on different DNN structures, and validated its capacity in
enhancing DNN training efficiency and performance.

提出了一种优化深度神经网络训练效果的方法，通过参数预测来改进模型性能，通过实验证实了该方法在不同的深度神经网络结构上提高了训练效率和性能。

通过线性预测提高深度神经网络训练效率和性能

Enhancing Deep Neural Network Training Efficiency and Performance  through Linear Prediction

Two potential bottlenecks on the expressiveness of recurrent neural networks
(RNNs) are their ability to store information about the task in their
parameters, and to store information about the input history in their units. We
show experimentally that all common RNN architectures achieve nearly the same
per-task and per-unit capacity bounds with careful training, for a variety of
tasks and stacking depths. They can store an amount of task information which
is linear in the number of parameters, and is approximately 5 bits per
parameter. They can additionally store approximately one real number from their
input history per hidden unit. We further find that for several tasks it is the
per-task parameter capacity bound that determines performance. These results
suggest that many previous results comparing RNN architectures are driven
primarily by differences in training effectiveness, rather than differences in
capacity. Supporting this observation, we compare training difficulty for
several architectures, and show that vanilla RNNs are far more difficult to
train, yet have slightly higher capacity. Finally, we propose two novel RNN
architectures, one of which is easier to train than the LSTM or GRU for deeply
stacked architectures.

通过实验证实了，所有常见的递归神经网络在仔细训练后达到了近乎相同的任务能力和单元容量边界，而许多以前的结果比较不同的递归神经网络结构主要是由于训练效果的差异而不是由于容量的差异驱动的。同时，作者还提出了两个新的递归神经网络结构，其中一个对于深度堆叠结构的训练比 LSTM 或 GRU 更容易。