Vision Transformers (ViTs) are normally regarded as a stack of transformer
layers. In this work, we propose a novel view of ViTs showing that they can be
seen as ensemble networks containing multiple parallel paths with different
lengths. Specifically, we equivalently transform the traditional cascade of
multi-head self-attention (MSA) and feed-forward network (FFN) into three
parallel paths in each transformer layer. Then, we utilize the identity
connection in our new transformer form and further transform the ViT into an
explicit multi-path ensemble network. From the new perspective, these paths
perform two functions: the first is to provide the feature for the classifier
directly, and the second is to provide the lower-level feature representation
for subsequent longer paths. We investigate the influence of each path for the
final prediction and discover that some paths even pull down the performance.
Therefore, we propose the path pruning and EnsembleScale skills for
improvement, which cut out the underperforming paths and re-weight the ensemble
components, respectively, to optimize the path combination and make the short
paths focus on providing high-quality representation for subsequent paths. We
also demonstrate that our path combination strategies can help ViTs go deeper
and act as high-pass filters to filter out partial low-frequency signals. To
further enhance the representation of paths served for subsequent paths,
self-distillation is applied to transfer knowledge from the long paths to the
short paths. This work calls for more future research to explain and design
ViTs from new perspectives.

Vision Transformers（ViTs）可以被看做包含多条不同长度的平行路径的集合网络，通过路径修剪和自我蒸馏技术来优化路径组合，提供高质量的特征表示，并且充当高通滤波器以过滤部分低频信号。