The growing popularity of Vision Transformers as the go-to models for image
classification has led to an explosion of architectural modifications claiming
to be more efficient than the original ViT. However, a wide diversity of
experimental conditions prevents a fair comparison between all of them, based
solely on their reported results. To address this gap in comparability, we
conduct a comprehensive analysis of more than 30 models to evaluate the
efficiency of vision transformers and related architectures, considering
various performance metrics. Our benchmark provides a comparable baseline
across the landscape of efficiency-oriented transformers, unveiling a plethora
of surprising insights. For example, we discover that ViT is still Pareto
optimal across multiple efficiency metrics, despite the existence of several
alternative approaches claiming to be more efficient. Results also indicate
that hybrid attention-CNN models fare particularly well when it comes to low
inference memory and number of parameters, and also that it is better to scale
the model size, than the image size. Furthermore, we uncover a strong positive
correlation between the number of FLOPS and the training memory, which enables
the estimation of required VRAM from theoretical measurements alone.
Thanks to our holistic evaluation, this study offers valuable insights for
practitioners and researchers, facilitating informed decisions when selecting
models for specific applications. We publicly release our code and data at
this https URL

对视觉 Transformer 及相关架构的效率进行了综合分析，揭示了一系列有趣的见解，例如发现 ViT 在多个效率度量标准上仍然是最佳选择，同时低推理内存和参数数量时，混合注意力 - CNN 模型表现良好，模型大小的缩放比图像大小更为重要，FLOPS 与训练内存之间存在强正相关性。