We present a novel Transformer-based network architecture for instance-aware
image-to-image translation, dubbed InstaFormer, to effectively integrate
global- and instance-level information. By considering extracted content
features from an image as tokens, our networks discover global consensus of
content features by considering context information through a self-attention
module in Transformers. By augmenting such tokens with an instance-level
feature extracted from the content feature with respect to bounding box
information, our framework is capable of learning an interaction between object
instances and the global image, thus boosting the instance-awareness. We
replace layer normalization (LayerNorm) in standard Transformers with adaptive
instance normalization (AdaIN) to enable a multi-modal translation with style
codes. In addition, to improve the instance-awareness and translation quality
at object regions, we present an instance-level content contrastive loss
defined between input and translated image. We conduct experiments to
demonstrate the effectiveness of our InstaFormer over the latest methods and
provide extensive ablation studies.

提出了一种新型的基于 Transformer 的网络结构 InstaFormer，用于实例感知的图像到图像的转换，可以有效地整合全局和实例级信息，通过自注意力模块在 Transformers 中考虑上下文信息，通过将通过边界框信息从内容特征中提取的实例级特征与这些标记相结合，我们的框架能够学习对象实例和全局图像之间的互动，从而提高实例感知能力，同时在标准 Transformer 中使用自适应实例归一化（AdaIN）代替层规范化（LayerNorm），以启用具有风格编码的多模式翻译。另外，为了提高实例感知能力和物体区域的翻译质量，介绍了一种实例级内容对比损失，定义了输入图像和翻译图像之间的对比损失。实验表明，与最新方法相比，我们的 InstaFormer 具有更好的性能。