The emergence of vision transformers (ViTs) in image classification has
shifted the methodologies for visual representation learning. In particular,
ViTs learn visual representation at full receptive field per layer across all
the image patches, in comparison to the increasing receptive fields of CNNs
across layers and other alternatives (e.g., large kernels and atrous
convolution). In this work, for the first time we explore the global context
learning potentials of ViTs for dense visual prediction (e.g., semantic
segmentation). Our motivation is that through learning global context at full
receptive field layer by layer, ViTs may capture stronger long-range dependency
information, critical for dense prediction tasks. We first demonstrate that
encoding an image as a sequence of patches, a vanilla ViT without local
convolution and resolution reduction can yield stronger visual representation
for semantic segmentation. For example, our model, termed as SEgmentation
TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the
test leaderboard on the day of submission) and Pascal Context (55.83% mIoU),
and performs competitively on Cityscapes. For tackling general dense visual
prediction tasks in a cost-effective manner, we further formulate a family of
Hierarchical Local-Global (HLG) Transformers, characterized by local attention
within windows and global-attention across windows in a pyramidal architecture.
Extensive experiments show that our methods achieve appealing performance on a
variety of dense prediction tasks (e.g., object detection and instance
segmentation and semantic segmentation) as well as image classification. Our
code and models are available at this https URL

本研究首次探索视觉变换器（ViT）的全局上下文学习潜力以实现密集视觉预测，编码图像作为一系列补丁并学习全局上下文，使 ViT 可以捕获更强的长距离依赖信息，提出的分割转换器（SETR）在 ADE20K 和 Pascal Context 数据集上取得了出色表现，同时提出了一系列分层本地 - 全局转换器，架构由局部关注（在窗口内）和全局关注（跨窗口）构成，对于各种密集预测任务具有吸引人的性能