Real-time on-device continual learning is needed for new applications such as
home robots, user personalization on smartphones, and augmented/virtual reality
headsets. However, this setting poses unique challenges: embedded devices have
limited memory and compute capacity and conventional machine learning models
suffer from catastrophic forgetting when updated on non-stationary data
streams. While several online continual learning models have been developed,
their effectiveness for embedded applications has not been rigorously studied.
In this paper, we first identify criteria that online continual learners must
meet to effectively perform real-time, on-device learning. We then study the
efficacy of several online continual learning methods when used with mobile
neural networks. We measure their performance, memory usage, compute
requirements, and ability to generalize to out-of-domain inputs.

本文研究了使用移动神经网络进行实时嵌入式设备上的在线连续学习的效果，通过测量它们的性能、内存使用、计算要求和对域外输入的泛化能力，验证了实时在设备上执行学习的有效性。

嵌入式设备的在线持续学习

Online Continual Learning for Embedded Devices

Non-Local (NL) blocks have been widely studied in various vision tasks.
However, it has been rarely explored to embed the NL blocks in mobile neural
networks, mainly due to the following challenges: 1) NL blocks generally have
heavy computation cost which makes it difficult to be applied in applications
where computational resources are limited, and 2) it is an open problem to
discover an optimal configuration to embed NL blocks into mobile neural
networks. We propose AutoNL to overcome the above two obstacles. Firstly, we
propose a Lightweight Non-Local (LightNL) block by squeezing the transformation
operations and incorporating compact features. With the novel design choices,
the proposed LightNL block is 400x computationally cheaper} than its
conventional counterpart without sacrificing the performance. Secondly, by
relaxing the structure of the LightNL block to be differentiable during
training, we propose an efficient neural architecture search algorithm to learn
an optimal configuration of LightNL blocks in an end-to-end manner. Notably,
using only 32 GPU hours, the searched AutoNL model achieves 77.7% top-1
accuracy on ImageNet under a typical mobile setting (350M FLOPs), significantly
outperforming previous mobile models including MobileNetV2 (+5.7%), FBNet
(+2.8%) and MnasNet (+2.1%). Code and models are available at
this https URL

本文提出了 AutoNL 算法解决了 Non-Local blocks 难以嵌入移动神经网络和配置不优等问题。AutoNL 设计了轻量级的 Lightweight Non-Local block 和一种有效的神经架构搜索算法，仅使用 32 个 GPU 小时即在 ImageNet 数据集的 350M FLOPs 的移动设定下实现了 77.7％的准确率，显著优于 MobileNetV2，FBNet 和 MnasNet 等先前的移动模型。