Fashion illustration is a crucial medium for designers to convey their
creative vision and transform design concepts into tangible representations
that showcase the interplay between clothing and the human body. In the context
of fashion design, computer vision techniques have the potential to enhance and
streamline the design process. Departing from prior research primarily focused
on virtual try-on, this paper tackles the task of multimodal-conditioned
fashion image editing. Our approach aims to generate human-centric fashion
images guided by multimodal prompts, including text, human body poses, garment
sketches, and fabric textures. To address this problem, we propose extending
latent diffusion models to incorporate these multiple modalities and modifying
the structure of the denoising network, taking multimodal prompts as input. To
condition the proposed architecture on fabric textures, we employ textual
inversion techniques and let diverse cross-attention layers of the denoising
network attend to textual and texture information, thus incorporating different
granularity conditioning details. Given the lack of datasets for the task, we
extend two existing fashion datasets, Dress Code and VITON-HD, with multimodal
annotations. Experimental evaluations demonstrate the effectiveness of our
proposed approach in terms of realism and coherence concerning the provided
multimodal inputs.

该论文探讨了多模态条件下的时尚图像编辑任务，提出了一种基于多模态指导的生成方法，利用时尚设计的文本、人体姿势、服装草图和面料纹理等多模态提示生成以人为中心的时尚图像。实验证明了提出方法在现实感和一致性方面的有效性。

时尚图像编辑的多模态条件化潜扩散模型

Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing

Fashion illustration is used by designers to communicate their vision and to
bring the design idea from conceptualization to realization, showing how
clothes interact with the human body. In this context, computer vision can thus
be used to improve the fashion design process. Differently from previous works
that mainly focused on the virtual try-on of garments, we propose the task of
multimodal-conditioned fashion image editing, guiding the generation of
human-centric fashion images by following multimodal prompts, such as text,
human body poses, and garment sketches. We tackle this problem by proposing a
new architecture based on latent diffusion models, an approach that has not
been used before in the fashion domain. Given the lack of existing datasets
suitable for the task, we also extend two existing fashion datasets, namely
Dress Code and VITON-HD, with multimodal annotations collected in a
semi-automatic manner. Experimental results on these new datasets demonstrate
the effectiveness of our proposal, both in terms of realism and coherence with
the given multimodal inputs. Source code and collected multimodal annotations
will be publicly released at:
this https URL

本文介绍了一种基于 latent diffusion models 的新型架构，通过多模态提示（文本、人体姿势和服装草图）引导生成人体中心时尚图像，同时扩展了两个已有的时尚数据集，证明了该方法的有效性。