Self-attention based models such as vision transformers (ViTs) have emerged
as a very competitive architecture alternative to convolutional neural networks
(CNNs) in computer vision. Despite increasingly stronger variants with
ever-higher recognition accuracies, due to the quadratic complexity of
self-attention, existing ViTs are typically demanding in computation and model
size. Although several successful design choices (e.g., the convolutions and
hierarchical multi-stage structure) of prior CNNs have been reintroduced into
recent ViTs, they are still not sufficient to meet the limited resource
requirements of mobile devices. This motivates a very recent attempt to develop
light ViTs based on the state-of-the-art MobileNet-v2, but still leaves a
performance gap behind. In this work, pushing further along this under-studied
direction we introduce EdgeViTs, a new family of light-weight ViTs that, for
the first time, enable attention-based vision models to compete with the best
light-weight CNNs in the tradeoff between accuracy and on-device efficiency.
This is realized by introducing a highly cost-effective local-global-local
(LGL) information exchange bottleneck based on optimal integration of
self-attention and convolutions. For device-dedicated evaluation, rather than
relying on inaccurate proxies like the number of FLOPs or parameters, we adopt
a practical approach of focusing directly on on-device latency and, for the
first time, energy efficiency. Specifically, we show that our models are
Pareto-optimal when both accuracy-latency and accuracy-energy trade-offs are
considered, achieving strict dominance over other ViTs in almost all cases and
competing with the most efficient CNNs. Code is available at
this https URL

通过引入高度成本效益的局部全局局部（LGL）信息交换瓶颈，结合最优的自注意力和卷积，我们引入了 EdgeViTs，这是一种新的轻便 ViTs 家族，它们能够在准确性和设备效率之间的权衡中与最佳轻量级 CNNs 竞争，并优于其他 ViTs 几乎在所有情况下，证实了模型是帕累托最优的。