Due to the high price and heavy energy consumption of GPUs, deploying deep
models on IoT devices such as microcontrollers makes significant contributions
for ecological AI. Conventional methods successfully enable convolutional
neural network inference of high resolution images on microcontrollers, while
the framework for vision transformers that achieve the state-of-the-art
performance in many vision applications still remains unexplored. In this
paper, we propose a hardware-algorithm co-optimizations method called MCUFormer
to deploy vision transformers on microcontrollers with extremely limited
memory, where we jointly design transformer architecture and construct the
inference operator library to fit the memory resource constraint. More
specifically, we generalize the one-shot network architecture search (NAS) to
discover the optimal architecture with highest task performance given the
memory budget from the microcontrollers, where we enlarge the existing search
space of vision transformers by considering the low-rank decomposition
dimensions and patch resolution for memory reduction. For the construction of
the inference operator library of vision transformers, we schedule the memory
buffer during inference through operator integration, patch embedding
decomposition, and token overwriting, allowing the memory buffer to be fully
utilized to adapt to the forward pass of the vision transformer. Experimental
results demonstrate that our MCUFormer achieves 73.62\% top-1 accuracy on
ImageNet for image classification with 320KB memory on STM32F746
microcontroller. Code is available at this https URL

本文提出了一种名为 MCUFormer 的硬件算法协同优化方法，用于在内存极为有限的微控制器上部署视觉变换器，实现图像分类任务，通过考虑降维尺寸和补丁分辨率来扩展视觉变换器的搜索空间，并通过操作符集成、补丁嵌入分解和令牌覆写等方法来构建视觉变换器的推理操作库，实验结果表明 MCUFormer 在 STM32F746 微控制器上实现了 320KB 内存的 ImageNet 图像分类任务的 73.62％的 Top-1 准确率。