Developing deep learning models on tiny devices (e.g. Microcontroller units, MCUs) has attracted much attention in various embedded IoT applications. However, it is challenging to efficiently design and deploy recent advanced models (e.g. transformers) on tiny devices due to their severe hardware resource constraints. In this work, we propose TinyFormer, a framework specifically designed to develop and deploy resource-efficient transformers on MCUs. TinyFormer mainly consists of SuperNAS, SparseNAS and SparseEngine. Separately, SuperNAS aims to search for an appropriate supernet from a vast search space. SparseNAS evaluates the best sparse single-path model including transformer architecture from the identified supernet. Finally, SparseEngine efficiently deploys the searched sparse models onto MCUs. To the best of our knowledge, SparseEngine is the first deployment framework capable of performing inference of sparse models with transformer on MCUs. Evaluation results on the CIFAR-10 dataset demonstrate that TinyFormer can develop efficient transformers with an accuracy of $96.1\%$ while adhering to hardware constraints of $1$MB storage and $320$KB memory. Additionally, TinyFormer achieves significant speedups in sparse inference, up to $12.2\times$, when compared to the CMSIS-NN library. TinyFormer is believed to bring powerful transformers into TinyML scenarios and greatly expand the scope of deep learning applications.

本文提出了TinyFormer，这是一个专门设计用于在微控制器单元上开发和部署资源高效的transformers的框架，通过SuperNAS、SparseNAS和SparseEngine的组合，实现了在MCUs上高效部署sparse models以及稀疏推理。评估结果表明，TinyFormer能够以96.1%的准确率开发有效的transformers，并在稀疏推理上相对于CMSIS-NN库实现了高达12.2倍的加速。TinyFormer将强大的transformers引入了TinyML场景，极大地扩展了深度学习应用领域。

TinyFormer：小型设备上高效的Transformer设计与部署