Convolutional networks and vision transformers have different forms of
pairwise interactions, pooling across layers and pooling at the end of the
network. Does the latter really need to be different? As a by-product of
pooling, vision transformers provide spatial attention for free, but this is
most often of low quality unless self-supervised, which is not well studied. Is
supervision really the problem?
In this work, we develop a generic pooling framework and then we formulate a
number of existing methods as instantiations. By discussing the properties of
each group of methods, we derive SimPool, a simple attention-based pooling
mechanism as a replacement of the default one for both convolutional and
transformer encoders. We find that, whether supervised or self-supervised, this
improves performance on pre-training and downstream tasks and provides
attention maps delineating object boundaries in all cases. One could thus call
SimPool universal. To our knowledge, we are the first to obtain attention maps
in supervised transformers of at least as good quality as self-supervised,
without explicit losses or modifying the architecture. Code at:
this https URL

我们通过提出一种通用的池化框架，将卷积网络和视觉变换器的默认池化机制替换为简单的基于注意力的池化机制 SimPool，从而改善了预训练和下游任务的性能，并在所有情况下提供了能够描绘物体边界的注意力分布图。