Transformers have become the standard in state-of-the-art vision
architectures, achieving impressive performance on both image-level and dense
pixelwise tasks. However, training vision transformers for high-resolution
pixelwise tasks has a prohibitive cost. Typical solutions boil down to
hierarchical architectures, fast and approximate attention, or training on
low-resolution crops. This latter solution does not constrain architectural
choices, but it leads to a clear performance drop when testing at resolutions
significantly higher than that used for training, thus requiring ad-hoc and
slow post-processing schemes. In this paper, we propose a novel strategy for
efficient training and inference of high-resolution vision transformers: the
key principle is to mask out most of the high-resolution inputs during
training, keeping only N random windows. This allows the model to learn local
interactions between tokens inside each window, and global interactions between
tokens from different windows. As a result, the model can directly process the
high-resolution input at test time without any special trick. We show that this
strategy is effective when using relative positional embedding such as rotary
embeddings. It is 4 times faster to train than a full-resolution network, and
it is straightforward to use at test time compared to existing approaches. We
apply this strategy to the dense monocular task of semantic segmentation, and
find that a simple setting with 2 windows performs best, hence the name of our
method: Win-Win. To demonstrate the generality of our contribution, we further
extend it to the binocular task of optical flow, reaching state-of-the-art
performance on the Spring benchmark that contains Full-HD images with an
inference time an order of magnitude faster than the best competitor.

高效训练和推理高分辨率视觉 Transformer 模型的新策略是只保留 N 个随机窗口的高分辨率输入，通过学习窗口内的标记间的局部交互和窗口间的全局交互，模型能够直接处理高分辨率输入。这种策略在使用相对位置嵌入（如旋转嵌入）时表现出了显著的有效性，训练速度是全分辨率网络的 4 倍，而且与现有方法相比使用起来更加简单。将此策略应用于语义分割等密集视觉任务中，发现使用 2 个窗口的简单设置效果最佳，因此命名为 Win-Win 方法。将此策略进一步扩展到光流等双目任务中，达到了在 Spring 基准测试中超越最佳竞争对手一个数量级更快推理时间的最先进性能。