Multi-scale learning is central to semantic segmentation. We visualize the
effective receptive field (ERF) of canonical multi-scale representations and
point out two risks in learning them: scale inadequacy and field inactivation.
A novel multi-scale learner, varying window attention (VWA), is presented to
address these issues. VWA leverages the local window attention (LWA) and
disentangles LWA into the query window and context window, allowing the
context's scale to vary for the query to learn representations at multiple
scales. However, varying the context to large-scale windows (enlarging ratio R)
can significantly increase the memory footprint and computation cost (R^2 times
larger than LWA). We propose a simple but professional re-scaling strategy to
zero the extra induced cost without compromising performance. Consequently, VWA
uses the same cost as LWA to overcome the receptive limitation of the local
window. Furthermore, depending on VWA and employing various MLPs, we introduce
a multi-scale decoder (MSD), VWFormer, to improve multi-scale representations
for semantic segmentation. VWFormer achieves efficiency competitive with the
most compute-friendly MSDs, like FPN and MLP decoder, but performs much better
than any MSDs. For instance, using nearly half of UPerNet's computation,
VWFormer outperforms it by 1.0%-2.5% mIoU on ADE20K. With little extra
overhead, ~10G FLOPs, Mask2Former armed with VWFormer improves by 1.0%-1.3%.

多尺度学习在语义分割中起着重要作用。本文针对多尺度学习过程中的不足提出了一种新型多尺度学习器，即变窗关注（VWA），它利用了本地窗口关注（LWA）并将其分解为查询窗口和上下文窗口，使得上下文的尺度可以变化以便查询窗口学习多尺度的表示。此外，作者还介绍了基于 VWA 的多尺度解码器（MSD），VWFormer，来提高语义分割的多尺度表示效果。VWFormer 在与其他计算友好的 MSD 算法相比的情况下，具有竞争力的计算效率，但是性能要比其他算法好得多。