Transformers have been extensively studied in medical image segmentation to
build pairwise long-range dependence. Yet, relatively limited well-annotated
medical image data makes transformers struggle to extract diverse global
features, resulting in attention collapse where attention maps become similar
or even identical. Comparatively, convolutional neural networks (CNNs) have
better convergence properties on small-scale training data but suffer from
limited receptive fields. Existing works are dedicated to exploring the
combinations of CNN and transformers while ignoring attention collapse, leaving
the potential of transformers under-explored. In this paper, we propose to
build CNN-style Transformers (ConvFormer) to promote better attention
convergence and thus better segmentation performance. Specifically, ConvFormer
consists of pooling, CNN-style self-attention (CSA), and convolutional
feed-forward network (CFFN) corresponding to tokenization, self-attention, and
feed-forward network in vanilla vision transformers. In contrast to positional
embedding and tokenization, ConvFormer adopts 2D convolution and max-pooling
for both position information preservation and feature size reduction. In this
way, CSA takes 2D feature maps as inputs and establishes long-range dependency
by constructing self-attention matrices as convolution kernels with adaptive
sizes. Following CSA, 2D convolution is utilized for feature refinement through
CFFN. Experimental results on multiple datasets demonstrate the effectiveness
of ConvFormer working as a plug-and-play module for consistent performance
improvement of transformer-based frameworks. Code is available at
this https URL

通过构建 CNN 式 Transformer（ConvFormer）来提升更好的注意力收敛，从而获得更好的分割性能。