Vision Transformer (ViT) architectures are becoming increasingly popular and
widely employed to tackle computer vision applications. Their main feature is
the capacity to extract global information through the self-attention
mechanism, outperforming earlier convolutional neural networks. However, ViT
deployment and performance have grown steadily with their size, number of
trainable parameters, and operations. Furthermore, self-attention's
computational and memory cost quadratically increases with the image
resolution. Generally speaking, it is challenging to employ these architectures
in real-world applications due to many hardware and environmental restrictions,
such as processing and computational capabilities. Therefore, this survey
investigates the most efficient methodologies to ensure sub-optimal estimation
performances. More in detail, four efficient categories will be analyzed:
compact architecture, pruning, knowledge distillation, and quantization
strategies. Moreover, a new metric called Efficient Error Rate has been
introduced in order to normalize and compare models' features that affect
hardware devices at inference time, such as the number of parameters, bits,
FLOPs, and model size. Summarizing, this paper firstly mathematically defines
the strategies used to make Vision Transformer efficient, describes and
discusses state-of-the-art methodologies, and analyzes their performances over
different application scenarios. Toward the end of this paper, we also discuss
open challenges and promising research directions.

这篇文章首先数学上定义了使 Vision Transformer 高效的策略，描述并讨论了最先进的方法学，并分析了它们在不同应用场景下的性能。

高效视觉变换器的调研：算法、技术和性能评测

A survey on efficient vision transformers: algorithms, techniques, and  performance benchmarking

Automated design of neural network architectures tailored for a specific task
is an extremely promising, albeit inherently difficult, avenue to explore.
While most results in this domain have been achieved on image classification
and language modelling problems, here we concentrate on dense per-pixel tasks,
in particular, semantic image segmentation using fully convolutional networks.
In contrast to the aforementioned areas, the design choices of a fully
convolutional network require several changes, ranging from the sort of
operations that need to be used---e.g., dilated convolutions---to a solving of
a more difficult optimisation problem. In this work, we are particularly
interested in searching for high-performance compact segmentation
architectures, able to run in real-time using limited resources. To achieve
that, we intentionally over-parameterise the architecture during the training
time via a set of auxiliary cells that provide an intermediate supervisory
signal and can be omitted during the evaluation phase. The design of the
auxiliary cell is emitted by a controller, a neural network with the fixed
structure trained using reinforcement learning. More crucially, we demonstrate
how to efficiently search for these architectures within limited time and
computational budgets. In particular, we rely on a progressive strategy that
terminates non-promising architectures from being further trained, and on
Polyak averaging coupled with knowledge distillation to speed-up the
convergence. Quantitatively, in 8 GPU-days our approach discovers a set of
architectures performing on-par with state-of-the-art among compact models on
the semantic segmentation, pose estimation and depth prediction tasks. Code
will be made available here: this https URL

通过强化学习算法训练固定结构的神经网络控制器，利用辅助模块提供中间监督信号，设计并在有限时间和计算预算内高效搜索出在密集像素任务上性能高、结构紧凑的分割模型。在 8 GPU-days 内，模型在语义分割、姿势估计和深度预测任务上表现良好。