Tiny deep learning on microcontroller units (MCUs) is challenging due to the
limited memory size. We find that the memory bottleneck is due to the
imbalanced memory distribution in convolutional neural network (CNN) designs:
the first several blocks have an order of magnitude larger memory usage than
the rest of the network. To alleviate this issue, we propose a generic
patch-by-patch inference scheduling, which operates only on a small spatial
region of the feature map and significantly cuts down the peak memory. However,
naive implementation brings overlapping patches and computation overhead. We
further propose network redistribution to shift the receptive field and FLOPs
to the later stage and reduce the computation overhead. Manually redistributing
the receptive field is difficult. We automate the process with neural
architecture search to jointly optimize the neural architecture and inference
scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the
peak memory usage of existing networks by 4-8x. Co-designed with neural
networks, MCUNetV2 sets a record ImageNet accuracy on MCU (71.8%), and achieves
>90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2
also unblocks object detection on tiny devices, achieving 16.9% higher mAP on
Pascal VOC compared to the state-of-the-art result. Our study largely addressed
the memory bottleneck in tinyML and paved the way for various vision
applications beyond image classification.

该研究提出了一种基于分块的推理调度方法和神经网络重构技术，通过结合神经网络的设计和推理调度，实现了在微控制器单元上高效、准确的深度学习推理，并改善了已有网络的内存瓶颈问题。

MCUNetV2: 面向小型深度学习的高效内存补丁推理

MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

In the last few years, research and development on Deep Learning models and
techniques for ultra-low-power devices in a word, TinyML has mainly focused on
a train-then-deploy assumption, with static models that cannot be adapted to
newly collected data without cloud-based data collection and fine-tuning.
Latent Replay-based Continual Learning (CL) techniques[1] enable online,
serverless adaptation in principle, but so farthey have still been too
computation and memory-hungry for ultra-low-power TinyML devices, which are
typically based on microcontrollers. In this work, we introduce a HW/SW
platform for end-to-end CL based on a 10-core FP32-enabled parallel
ultra-low-power (PULP) processor. We rethink the baseline Latent Replay CL
algorithm, leveraging quantization of the frozen stage of the model and Latent
Replays (LRs) to reduce their memory cost with minimal impact on accuracy. In
particular, 8-bit compression of the LR memory proves to be almost lossless
(-0.26% with 3000LR) compared to the full-precision baseline implementation,
but requires 4x less memory, while 7-bit can also be used with an additional
minimal accuracy degradation (up to 5%). We also introduce optimized primitives
for forward and backward propagation on the PULP processor. Our results show
that by combining these techniques, continual learning can be achieved in
practice using less than 64MB of memory an amount compatible with embedding in
TinyML devices. On an advanced 22nm prototype of our platform, called VEGA, the
proposed solution performs onaverage 65x faster than a low-power STM32 L4
microcontroller, being 37x more energy efficient enough for a lifetime of 535h
when learning a new mini-batch of data once every minute.

本文提出了基于 PULP 处理器的深度学习端到端持续学习硬件 / 软件平台，利用量化和潜在重播来优化内存成本，以实现采用小型设备的实践多样性更新。

一个用于设备上连续学习的 TinyML 平台，具备量化的潜在回放功能

A TinyML Platform for On-Device Continual Learning with Quantized Latent  Replays

Executing machine learning workloads locally on resource constrained
microcontrollers (MCUs) promises to drastically expand the application space of
IoT. However, so-called TinyML presents severe technical challenges, as deep
neural network inference demands a large compute and memory budget. To address
this challenge, neural architecture search (NAS) promises to help design
accurate ML models that meet the tight MCU memory, latency and energy
constraints. A key component of NAS algorithms is their latency/energy model,
i.e., the mapping from a given neural network architecture to its inference
latency/energy on an MCU. In this paper, we observe an intriguing property of
NAS search spaces for MCU model design: on average, model latency varies
linearly with model operation (op) count under a uniform prior over models in
the search space. Exploiting this insight, we employ differentiable NAS (DNAS)
to search for models with low memory usage and low op count, where op count is
treated as a viable proxy to latency. Experimental results validate our
methodology, yielding our MicroNet models, which we deploy on MCUs using
Tensorflow Lite Micro, a standard open-source NN inference runtime widely used
in the TinyML community. MicroNets demonstrate state-of-the-art results for all
three TinyMLperf industry-standard benchmark tasks: visual wake words, audio
keyword spotting, and anomaly detection. Models and training scripts can be
found at github.com/ARM-software/ML-zoo.

本文介绍了机器学习在资源受限的微控制器上的应用，使用神经结构搜索算法来设计满足内存、延迟和能耗限制的模型，并通过可微分神经结构搜索算法搜索出了具有低内存使用和低操作计数的 MicroNet 模型，验证该文述方法在视觉唤醒词、音频关键词检测和异常检测等领域的优越性。