Transformer with self-attention has led to the revolutionizing of natural
language processing field, and recently inspires the emergence of
Transformer-style architecture design with competitive results in numerous
computer vision tasks. Nevertheless, most of existing designs directly employ
self-attention over a 2D feature map to obtain the attention matrix based on
pairs of isolated queries and keys at each spatial location, but leave the rich
contexts among neighbor keys under-exploited. In this work, we design a novel
Transformer-style module, i.e., Contextual Transformer (CoT) block, for visual
recognition. Such design fully capitalizes on the contextual information among
input keys to guide the learning of dynamic attention matrix and thus
strengthens the capacity of visual representation. Technically, CoT block first
contextually encodes input keys via a $3\times3$ convolution, leading to a
static contextual representation of inputs. We further concatenate the encoded
keys with input queries to learn the dynamic multi-head attention matrix
through two consecutive $1\times1$ convolutions. The learnt attention matrix is
multiplied by input values to achieve the dynamic contextual representation of
inputs. The fusion of the static and dynamic contextual representations are
finally taken as outputs. Our CoT block is appealing in the view that it can
readily replace each $3\times3$ convolution in ResNet architectures, yielding a
Transformer-style backbone named as Contextual Transformer Networks (CoTNet).
Through extensive experiments over a wide range of applications (e.g., image
recognition, object detection and instance segmentation), we validate the
superiority of CoTNet as a stronger backbone. Source code is available at
https://github.com/JDAI-CV/CoTNet.

该论文提出了一种新颖的 Transformer 风格模块，即 Contextual Transformer (CoT) 块，它完全利用输入键之间的情境信息来指导动态注意力矩阵的学习，从而增强了视觉表示能力，可作为更强大的骨干网。