Existing generative adversarial network (GAN) based conditional image
generative models typically produce fixed output for the same conditional
input, which is unreasonable for highly subjective tasks, such as large-mask
image inpainting or style transfer. On the other hand, GAN-based diverse image
generative methods require retraining/fine-tuning the network or designing
complex noise injection functions, which is computationally expensive,
task-specific, or struggle to generate high-quality results. Given that many
deterministic conditional image generative models have been able to produce
high-quality yet fixed results, we raise an intriguing question: is it possible
for pre-trained deterministic conditional image generative models to generate
diverse results without changing network structures or parameters? To answer
this question, we re-examine the conditional image generation tasks from the
perspective of adversarial attack and propose a simple and efficient plug-in
projected gradient descent (PGD) like method for diverse and controllable image
generation. The key idea is attacking the pre-trained deterministic generative
models by adding a micro perturbation to the input condition. In this way,
diverse results can be generated without any adjustment of network structures
or fine-tuning of the pre-trained models. In addition, we can also control the
diverse results to be generated by specifying the attack direction according to
a reference text or image. Our work opens the door to applying adversarial
attack to low-level vision tasks, and experiments on various conditional image
generation tasks demonstrate the effectiveness and superiority of the proposed
method.

基于预训练的确定性条件图像生成模型的网络结构或参数不变，我们通过添加微小扰动攻击输入条件，提出了一种简单高效的插件投影梯度下降（PGD）方法来生成多样且可控的图像，从而为低层视觉任务应用对抗攻击打开了新的可能性。

针对确定性条件图像生成模型的攻击：多样化和可控生成

Attack Deterministic Conditional Image Generative Models for Diverse and  Controllable Generation

We introduce a new method for diverse foreground generation with explicit
control over various factors. Existing image inpainting based foreground
generation methods often struggle to generate diverse results and rarely allow
users to explicitly control specific factors of variation (e.g., varying the
facial identity or expression for face inpainting results). We leverage
contrastive learning with latent codes to generate diverse foreground results
for the same masked input. Specifically, we define two sets of latent codes,
where one controls a pre-defined factor (``known''), and the other controls the
remaining factors (``unknown''). The sampled latent codes from the two sets
jointly bi-modulate the convolution kernels to guide the generator to
synthesize diverse results. Experiments demonstrate the superiority of our
method over state-of-the-arts in result diversity and generation
controllability.

本研究提出了一种新方法，使用对比学习和潜在编码来进行前景生成，实现了明确控制各种因素，获得多样性和可控性的生成结果，比现有技术更优秀。

对比学习用于多样性解缠前景生成

Contrastive Learning for Diverse Disentangled Foreground Generation

Given an incomplete image without additional constraint, image inpainting
natively allows for multiple solutions as long as they appear plausible.
Recently, multiplesolution inpainting methods have been proposed and shown the
potential of generating diverse results. However, these methods have difficulty
in ensuring the quality of each solution, e.g. they produce distorted structure
and/or blurry texture. We propose a two-stage model for diverse inpainting,
where the first stage generates multiple coarse results each of which has a
different structure, and the second stage refines each coarse result separately
by augmenting texture. The proposed model is inspired by the hierarchical
vector quantized variational auto-encoder (VQ-VAE), whose hierarchical
architecture isentangles structural and textural information. In addition, the
vector quantization in VQVAE enables autoregressive modeling of the discrete
distribution over the structural information. Sampling from the distribution
can easily generate diverse and high-quality structures, making up the first
stage of our model. In the second stage, we propose a structural attention
module inside the texture generation network, where the module utilizes the
structural information to capture distant correlations. We further reuse the
VQ-VAE to calculate two feature losses, which help improve structure coherence
and texture realism, respectively. Experimental results on CelebA-HQ, Places2,
and ImageNet datasets show that our method not only enhances the diversity of
the inpainting solutions but also improves the visual quality of the generated
multiple images. Code and models are available at:
this https URL

本文提出了一种基于 VQ-VAE 的两阶段模型，在第一阶段生成多个不同结构的粗略结果，第二阶段对于每个粗略结果进行纹理增强，其中纹理生成网络中的结构注意模块利用结构信息捕获远程相关性。这种方法不仅增强了图像修复解的多样性，还提高了生成的多张图像的视觉质量。

采用分层 VQ-VAE 对图像修复生成多样化结构

Generating Diverse Structure for Image Inpainting With Hierarchical  VQ-VAE

Many image-to-image translation problems are ambiguous, as a single input
image may correspond to multiple possible outputs. In this work, we aim to
model a \emph{distribution} of possible outputs in a conditional generative
modeling setting. The ambiguity of the mapping is distilled in a
low-dimensional latent vector, which can be randomly sampled at test time. A
generator learns to map the given input, combined with this latent code, to the
output. We explicitly encourage the connection between output and the latent
code to be invertible. This helps prevent a many-to-one mapping from the latent
code to the output during training, also known as the problem of mode collapse,
and produces more diverse results. We explore several variants of this approach
by employing different training objectives, network architectures, and methods
of injecting the latent code. Our proposed method encourages bijective
consistency between the latent encoding and output modes. We present a
systematic comparison of our method and other variants on both perceptual
realism and diversity.

利用条件生成模型和低维潜在向量，通过鼓励输出和潜在编码间的双射一致性的方法解决图像转换中一对多的映射问题，并产生更丰富多样的结果。