To facilitate the evolution of edge intelligence in ever-changing
environments, we study on-device incremental learning constrained in limited
computation resource in this paper. Current on-device training methods just
focus on efficient training without considering the catastrophic forgetting,
preventing the model getting stronger when continually exploring the world. To
solve this problem, a direct solution is to involve the existing incremental
learning mechanisms into the on-device training framework. Unfortunately, such
a manner cannot work well as those mechanisms usually introduce large
additional computational cost to the network optimization process, which would
inevitably exceed the memory capacity of the edge devices. To address this
issue, this paper makes an early effort to propose a simple but effective
edge-friendly incremental learning framework. Based on an empirical study on
the knowledge intensity of the kernel elements of the neural network, we find
that the center kernel is the key for maximizing the knowledge intensity for
learning new data, while freezing the other kernel elements would get a good
balance on the model's capacity for overcoming catastrophic forgetting. Upon
this finding, we further design a center-sensitive kernel optimization
framework to largely alleviate the cost of the gradient computation and
back-propagation. Besides, a dynamic channel element selection strategy is also
proposed to facilitate a sparse orthogonal gradient projection for further
reducing the optimization complexity, upon the knowledge explored from the new
task data. Extensive experiments validate our method is efficient and
effective, e.g., our method achieves average accuracy boost of 38.08% with even
less memory and approximate computation compared to existing on-device training
methods, indicating its significant potential for on-device incremental
learning.

为促进边缘智能在不断变化的环境中的发展，本文研究了在有限计算资源约束下的设备上的增量学习。与当前只关注高效训练而不考虑灾难性遗忘的方法不同，本文提出了一种简单而有效的面向边缘的增量学习框架，通过对神经网络的核心元素的知识强度的经验研究，发现中心核心对于学习新数据的知识强度最大化起关键作用，冻结其他核心元素可以在克服灾难性遗忘方面取得良好平衡。基于这一发现，我们进一步设计了一个便于梯度计算和反向传播的中心敏感核心优化框架，并提出了一种动态通道元素选择策略，以促进稀疏正交梯度投影以进一步减少优化复杂性。广泛的实验验证了我们的方法的高效性和有效性，例如，与现有的设备上训练方法相比，我们的方法在甚至更少的内存和近似计算的情况下实现了平均准确度提升 38.08%，表明其在设备上的增量学习方面具有显著潜力。