Motivated by the huge success of Transformers in the field of natural
language processing (NLP), Vision Transformers (ViTs) have been rapidly
developed and achieved remarkable performance in various computer vision tasks.
However, their huge model sizes and intensive computations hinder ViTs'
deployment on embedded devices, calling for effective model compression
methods, such as quantization. Unfortunately, due to the existence of
hardware-unfriendly and quantization-sensitive non-linear operations,
particularly {Softmax}, it is non-trivial to completely quantize all operations
in ViTs, yielding either significant accuracy drops or non-negligible hardware
costs. In response to challenges associated with \textit{standard ViTs}, we
focus our attention towards the quantization and acceleration for
\textit{efficient ViTs}, which not only eliminate the troublesome Softmax but
also integrate linear attention with low computational complexity, and propose
\emph{Trio-ViT} accordingly. Specifically, at the algorithm level, we develop a
{tailored post-training quantization engine} taking the unique activation
distributions of Softmax-free efficient ViTs into full consideration, aiming to
boost quantization accuracy. Furthermore, at the hardware level, we build an
accelerator dedicated to the specific Convolution-Transformer hybrid
architecture of efficient ViTs, thereby enhancing hardware efficiency.
Extensive experimental results consistently prove the effectiveness of our
Trio-ViT framework. {Particularly, we can gain up to
$\uparrow$$\mathbf{7.2}\times$ and $\uparrow$$\mathbf{14.6}\times$ FPS under
comparable accuracy over state-of-the-art ViT accelerators, as well as
$\uparrow$$\mathbf{5.9}\times$ and $\uparrow$$\mathbf{2.0}\times$ DSP
efficiency.} Codes will be released publicly upon acceptance.

通过量化和加速高效 ViTs 的方法，提出了 Trio-ViT 框架，该框架在可比准确度下获得了可观的 FPS 和 DSP 效率提升，解决了 Vision Transformers 在嵌入式设备上部署困难的问题。