The performance of face photo-sketch translation has improved a lot thanks to
deep neural networks. GAN based methods trained on paired images can produce
high-quality results under laboratory settings. Such paired datasets are,
however, often very small and lack diversity. Meanwhile, Cycle-GANs trained
with unpaired photo-sketch datasets suffer from the \emph{steganography}
phenomenon, which makes them not effective to face photos in the wild. In this
paper, we introduce a semi-supervised approach with a noise-injection strategy,
named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we
propose a {\em pseudo sketch feature} representation for each input photo
composed from a small reference set of photo-sketch pairs, and use the
resulting {\em pseudo pairs} to supervise a photo-to-sketch generator
$G_{p2s}$. The outputs of $G_{p2s}$ can in turn help to train a sketch-to-photo
generator $G_{s2p}$ in a self-supervised manner. This allows us to train
$G_{p2s}$ and $G_{s2p}$ using a small reference set of photo-sketch pairs
together with a large face photo dataset (without ground-truth sketches). For
the second problem, we show that the simple noise-injection strategy works well
to alleviate the \emph{steganography} effect in SCG and helps to produce more
reasonable sketch-to-photo results with less overfitting than fully supervised
approaches. Experiments show that SCG achieves competitive performance on
public benchmarks and superior results on photos in the wild.

本文介绍一种半监督方法，即带有噪声注入策略的 Semi-Cycle-GAN (SCG)，用于解决面部照片素描翻译中的配对数据集小且缺乏多样性以及来自不匹配照片 - 素描数据集所带来的隐写学现象等问题。实验证明 SCG 在公共基准测试中具有竞争性的性能，并且在野外照片上表现出优越的结果。

野外环境下面部照片 - 素描互译的半监督循环生成对抗网络

Semi-supervised Cycle-GAN for face photo-sketch translation in the wild

Deformable Image Registration (DIR) of MR and CT images is one of the most
challenging registration task, due to the inherent structural differences of
the modalities and the missing dense ground truth. Recently cycle Generative
Adversarial Networks (cycle-GANs) have been used to learn the intensity
relationship between these 2 modalities for unpaired brain data. Yet its
usefulness for DIR was not assessed.
In this study we evaluate the DIR performance for thoracic and abdominal
organs after synthesis by cycle-GAN. We show that geometric changes, which
differentiate the two populations (e.g. inhale vs. exhale), are readily
synthesized as well. This causes substantial problems for any application which
relies on spatial correspondences being preserved between the real and the
synthesized image (e.g. plan, segmentation, landmark propagation). To alleviate
this problem, we investigated reducing the spatial information provided to the
discriminator by decreasing the size of its receptive fields.
Image synthesis was learned from 17 unpaired subjects per modality.
Registration performance was evaluated with respect to manual segmentations of
11 structures for 3 subjects from the VISERAL challenge. State-of-the-art DIR
methods based on Normalized Mutual Information (NMI), Modality Independent
Neighborhood Descriptor (MIND) and their novel combination achieved a mean
segmentation overlap ratio of 76.7, 67.7, 76.9%, respectively. This dropped to
69.1% or less when registering images synthesized by cycle-GAN based on local
correlation, due to the poor performance on the thoracic region, where large
lung volume changes were synthesized. Performance for the abdominal region was
similar to that of CT-MRI NMI registration (77.4 vs. 78.8%) when using 3D
synthesizing MRIs (12 slices) and medium sized receptive fields for the
discriminator.

本研究通过用 cycle-GAN 学习两个模态之间的强度关系，改善了 CT 和 MR 图像的配准问题，并探讨了使用 cycle-GAN 合成的图像减少空间信息对模式对位的影响。