To enable learning on edge devices with fast convergence and low memory, we
present a novel backpropagation-free optimization algorithm dubbed Target
Projection Stochastic Gradient Descent (tpSGD). tpSGD generalizes direct random
target projection to work with arbitrary loss functions and extends target
projection for training recurrent neural networks (RNNs) in addition to
feedforward networks. tpSGD uses layer-wise stochastic gradient descent (SGD)
and local targets generated via random projections of the labels to train the
network layer-by-layer with only forward passes. tpSGD doesn't require
retaining gradients during optimization, greatly reducing memory allocation
compared to SGD backpropagation (BP) methods that require multiple instances of
the entire neural network weights, input/output, and intermediate results. Our
method performs comparably to BP gradient-descent within 5% accuracy on
relatively shallow networks of fully connected layers, convolutional layers,
and recurrent layers. tpSGD also outperforms other state-of-the-art
gradient-free algorithms in shallow models consisting of multi-layer
perceptrons, convolutional neural networks (CNNs), and RNNs with competitive
accuracy and less memory and time. We evaluate the performance of tpSGD in
training deep neural networks (e.g. VGG) and extend the approach to multi-layer
RNNs. These experiments highlight new research directions related to optimized
layer-based adaptor training for domain-shift using tpSGD at the edge.

提出了一种名为 tpSGD 的优化算法，该算法可以在边缘设备上进行学习，并且收敛速度快，具有低内存占用，并且可以训练任意的损失函数和反向传播神经网络，避免了梯度保留的需求，实现了内存优化。实验结果表明，该方法相对于其他梯度自由算法具有更好的性能表现。