Attention mechanisms have been very popular in deep neural networks, where
the Transformer architecture has achieved great success in not only natural
language processing but also visual recognition applications. Recently, a new
Transformer module, applying on batch dimension rather than spatial/channel
dimension, i.e., BatchFormer [18], has been introduced to explore sample
relationships for overcoming data scarcity challenges. However, it only works
with image-level representations for classification. In this paper, we devise a
more general batch Transformer module, BatchFormerV2, which further enables
exploring sample relationships for dense representation learning. Specifically,
when applying the proposed module, it employs a two-stream pipeline during
training, i.e., either with or without a BatchFormerV2 module, where the
batchformer stream can be removed for testing. Therefore, the proposed method
is a plug-and-play module and can be easily integrated into different vision
Transformers without any extra inference cost. Without bells and whistles, we
show the effectiveness of the proposed method for a variety of popular visual
recognition tasks, including image classification and two important dense
prediction tasks: object detection and panoptic segmentation. Particularly,
BatchFormerV2 consistently improves current DETR-based detection methods (e.g.,
DETR, Deformable-DETR, Conditional DETR, and SMCA) by over 1.3%. Code will be
made publicly available.

本文提出了一种新的 Batch Transformer 模块，即 BatchFormerV2，可以进一步探索密集表示学习，适用于各种常见的视觉识别任务，包括图像分类、目标检测和全景分割，改进了几种基于 DETR 的目标检测方法 (例如 DETR、Deformable-DETR、Conditional DETR 和 SMCA) 1.3% 以上。