Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are passively hidden in the CNN's increasing receptive fields or actively encoded by non-local convolution. Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales. To this end, we propose a fully active feature interaction across both space and scales, called Feature Pyramid Transformer (FPT). It transforms any feature pyramid into another feature pyramid of the same size but with richer contexts, by using three specially designed transformers in self-level, top-down, and bottom-up interaction fashion. FPT serves as a generic visual backbone with fair computational overhead. We conduct extensive experiments in both instance-level (i.e., object detection and instance segmentation) and pixel-level segmentation tasks, using various backbones and head networks, and observe consistent improvement over all the baselines and the state-of-the-art methods.

提出了一种名为Feature Pyramid Transformer（FPT）的全新特征交互方法，能够将任何特征金字塔转换为具有更多上下文信息的相同大小的特征金字塔，从而有效捕捉不同尺度中物体的非局部上下文特征。在实例级和像素级分割任务中，使用不同的骨干网络和头部网络进行了广泛实验，并观察到相对于所有基线和现有最先进方法均有持续的提高。

特征金字塔变换器