Vision Transformers (ViTs) have achieved overwhelming success, yet they
suffer from vulnerable resolution scalability, i.e., the performance drops
drastically when presented with input resolutions that are unseen during
training. We introduce, ResFormer, a framework that is built upon the seminal
idea of multi-resolution training for improved performance on a wide spectrum
of, mostly unseen, testing resolutions. In particular, ResFormer operates on
replicated images of different resolutions and enforces a scale consistency
loss to engage interactive information across different scales. More
importantly, to alternate among varying resolutions effectively, especially
novel ones in testing, we propose a global-local positional embedding strategy
that changes smoothly conditioned on input sizes. We conduct extensive
experiments for image classification on ImageNet. The results provide strong
quantitative evidence that ResFormer has promising scaling abilities towards a
wide range of resolutions. For instance, ResFormer-B-MR achieves a Top-1
accuracy of 75.86% and 81.72% when evaluated on relatively low and high
resolutions respectively (i.e., 96 and 640), which are 48% and 7.49% better
than DeiT-B. We also demonstrate, moreover, ResFormer is flexible and can be
easily extended to semantic segmentation, object detection and video action
recognition. Code is available at this https URL

ResFormer 框架通过多分辨率训练方法和全局 - 局部位置嵌入策略，在图像分类、目标检测、语义分割和视频动作识别等领域取得了较好的结果。