We propose a novel ECGAN for the challenging semantic image synthesis task.
Although considerable improvements have been achieved by the community in the
recent period, the quality of synthesized images is far from satisfactory due
to three largely unresolved challenges. 1) The semantic labels do not provide
detailed structural information, making it challenging to synthesize local
details and structures; 2) The widely adopted CNN operations such as
convolution, down-sampling, and normalization usually cause spatial resolution
loss and thus cannot fully preserve the original semantic information, leading
to semantically inconsistent results (e.g., missing small objects); 3) Existing
semantic image synthesis methods focus on modeling 'local' semantic information
from a single input semantic layout. However, they ignore 'global' semantic
information of multiple input semantic layouts, i.e., semantic cross-relations
between pixels across different input layouts. To tackle 1), we propose to use
the edge as an intermediate representation which is further adopted to guide
image generation via a proposed attention guided edge transfer module. To
tackle 2), we design an effective module to selectively highlight
class-dependent feature maps according to the original semantic layout to
preserve the semantic information. To tackle 3), inspired by current methods in
contrastive learning, we propose a novel contrastive learning method, which
aims to enforce pixel embeddings belonging to the same semantic class to
generate more similar image content than those from different classes. We
further propose a novel multi-scale contrastive learning method that aims to
push same-class features from different scales closer together being able to
capture more semantic relations by explicitly exploring the structures of
labeled pixels from multiple input semantic layouts from different scales.

提出了一种新颖的 ECGAN 用于具有挑战性的语义图像合成任务，通过使用边缘作为中间表示，并采用注意力引导的边缘传输模块进行图像生成，设计了一个有效的模块来选择性地突出显示根据原始语义布局的类依赖特征图，通过对比学习方法和多尺度对比学习方法，强化了同一语义类别的像素嵌入生成更相似的图像内容，并利用多个输入语义布局的标记像素结构来明确地探索更多的语义关系。