Transformer has achieved great success in computer vision, while how to split
patches in an image remains a problem. Existing methods usually use a
fixed-size patch embedding which might destroy the semantics of objects. To
address this problem, we propose a new Deformable Patch (DePatch) module which
learns to adaptively split the images into patches with different positions and
scales in a data-driven way rather than using predefined fixed patches. In this
way, our method can well preserve the semantics in patches. The DePatch module
can work as a plug-and-play module, which can easily be incorporated into
different transformers to achieve an end-to-end training. We term this
DePatch-embedded transformer as Deformable Patch-based Transformer (DPT) and
conduct extensive evaluations of DPT on image classification and object
detection. Results show DPT can achieve 81.9% top-1 accuracy on ImageNet
classification, and 43.7% box mAP with RetinaNet, 44.3% with Mask R-CNN on
MSCOCO object detection. Code has been made available at:
this https URL .

提出了一种新型的 Deformable Patch（DePatch）模块，可以自适应地将图像分割为具有不同位置和比例的补丁，从而更好地保留了补丁中的语义信息。将该模块加入 Transformer 中，在图像分类和目标检测等任务上进行广泛评估。