Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.

本研究提出了一种基于图像到图像翻译的新型无监督方法，通过类似于“将头发颜色变为黑色”的命令句子来改变给定图像的属性。我们的模型可以学习使用文本描述修改图像的视觉属性，而无需人工注释的数据集或文本描述。实验结果表明，我们的方法在两个大规模公共数据集上表现良好，这将为将文本或语音命令与视觉属性相结合的新研究方向铺平道路。

描述改变什么：一种文本引导的无监督图像到图像转换方法