Transformers have shown superior performance on various vision tasks. Their
large receptive field endows Transformer models with higher representation
power than their CNN counterparts. Nevertheless, simply enlarging the receptive
field also raises several concerns. On the one hand, using dense attention in
ViT leads to excessive memory and computational cost, and features can be
influenced by irrelevant parts that are beyond the region of interests. On the
other hand, the handcrafted attention adopted in PVT or Swin Transformer is
data agnostic and may limit the ability to model long-range relations. To solve
this dilemma, we propose a novel deformable multi-head attention module, where
the positions of key and value pairs in self-attention are adaptively allocated
in a data-dependent way. This flexible scheme enables the proposed deformable
attention to dynamically focus on relevant regions while maintains the
representation power of global attention. On this basis, we present Deformable
Attention Transformer (DAT), a general vision backbone efficient and effective
for visual recognition. We further build an enhanced version DAT++. Extensive
experiments show that our DAT++ achieves state-of-the-art results on various
visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0
MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.

通过引入一种新的可变形多头注意力模块，Deformable Attention Transformer（DAT）有效地解决了 Transformer 模型中存在的注意力范围过大和过于手工化的问题，从而提高视觉识别任务的性能。实验证明，DAT 在各种视觉识别基准测试中取得了最先进的结果，包括 85.9% 的 ImageNet 准确率，54.5 和 47.0 的 MS-COCO 实例分割 mAP，以及 51.5 的 ADE20K 语义分割 mIoU。