We revisit the existing excellent Transformers from the perspective of
practical application. Most of them are not even as efficient as the basic
ResNets series and deviate from the realistic deployment scenario. It may be
due to the current criterion to measure computation efficiency, such as FLOPs
or parameters is one-sided, sub-optimal, and hardware-insensitive. Thus, this
paper directly treats the TensorRT latency on the specific hardware as an
efficiency metric, which provides more comprehensive feedback involving
computational capacity, memory cost, and bandwidth. Based on a series of
controlled experiments, this work derives four practical guidelines for
TensorRT-oriented and deployment-friendly network design, e.g., early CNN and
late Transformer at stage-level, early Transformer and late CNN at block-level.
Accordingly, a family of TensortRT-oriented Transformers is presented,
abbreviated as TRT-ViT. Extensive experiments demonstrate that TRT-ViT
significantly outperforms existing ConvNets and vision Transformers with
respect to the latency/accuracy trade-off across diverse visual tasks, e.g.,
image classification, object detection and semantic segmentation. For example,
at 82.7% ImageNet-1k top-1 accuracy, TRT-ViT is 2.7$\times$ faster than CSWin
and 2.0$\times$ faster than Twins. On the MS-COCO object detection task,
TRT-ViT achieves comparable performance with Twins, while the inference speed
is increased by 2.8$\times$.

本文主要从实际应用的角度重新审视 Transformer，并将硬件延迟作为衡量计算效率的指标，提出一系列针对 TensorRT 的网络设计实践指南，并给出了一家 TRT-ViT 的网络模型，实验表明在不同的视觉任务中，TRT-ViT 在延迟 / 准确性的权衡方面显著优于现有的 ConvNets 和 Vision Transformers。