We present M&M VTO, a mix and match virtual try-on method that takes as input
multiple garment images, text description for garment layout and an image of a
person. An example input includes: an image of a shirt, an image of a pair of
pants, "rolled sleeves, shirt tucked in", and an image of a person. The output
is a visualization of how those garments (in the desired layout) would look
like on the given person. Key contributions of our method are: 1) a single
stage diffusion based model, with no super resolution cascading, that allows to
mix and match multiple garments at 1024x512 resolution preserving and warping
intricate garment details, 2) architecture design (VTO UNet Diffusion
Transformer) to disentangle denoising from person specific features, allowing
for a highly effective finetuning strategy for identity preservation (6MB model
per individual vs 4GB achieved with, e.g., dreambooth finetuning); solving a
common identity loss problem in current virtual try-on methods, 3) layout
control for multiple garments via text inputs specifically finetuned over
PaLI-3 for virtual try-on task. Experimental results indicate that M&M VTO
achieves state-of-the-art performance both qualitatively and quantitatively, as
well as opens up new opportunities for virtual try-on via language-guided and
multi-garment try-on.

M&M VTO 是一种多样式虚拟试穿方法，以多个服装图像、服装布局的文本描述和人物图像作为输入，并输出展示给定人物上穿指定布局的服装效果的可视化结果。

M&M VTO: 多款服装虚拟试穿和编辑

M&M VTO: Multi-Garment Virtual Try-On and Editing

Recently, text-to-image diffusion models have demonstrated impressive ability
to generate high-quality images conditioned on the textual input. However,
these models struggle to accurately adhere to textual instructions regarding
spatial layout information. While previous research has primarily focused on
aligning cross-attention maps with layout conditions, they overlook the impact
of the initialization noise on the layout guidance. To achieve better layout
control, we propose leveraging a spatial-aware initialization noise during the
denoising process. Specifically, we find that the inverted reference image with
finite inversion steps contains valuable spatial awareness regarding the
object's position, resulting in similar layouts in the generated images. Based
on this observation, we develop an open-vocabulary framework to customize a
spatial-aware initialization noise for each layout condition. Without modifying
other modules except the initialization noise, our approach can be seamlessly
integrated as a plug-and-play module within other training-free layout guidance
frameworks. We evaluate our approach quantitatively and qualitatively on the
available Stable Diffusion model and COCO dataset. Equipped with the
spatial-aware latent initialization, our method significantly improves the
effectiveness of layout guidance while preserving high-quality content.

通过利用空间感知的初始化噪声，我们提出了一种自定义空间感知初始化噪声的开放词汇框架，以有效提高布局引导的效果。

针对可控图像生成的空间感知潜变量初始化

Spatial-Aware Latent Initialization for Controllable Image Generation

Recent diffusion-based generators can produce high-quality images based only
on textual prompts. However, they do not correctly interpret instructions that
specify the spatial layout of the composition. We propose a simple approach
that can achieve robust layout control without requiring training or
fine-tuning the image generator. Our technique, which we call layout guidance,
manipulates the cross-attention layers that the model uses to interface textual
and visual information and steers the reconstruction in the desired direction
given, e.g., a user-specified layout. In order to determine how to best guide
attention, we study the role of different attention maps when generating images
and experiment with two alternative strategies, forward and backward guidance.
We evaluate our method quantitatively and qualitatively with several
experiments, validating its effectiveness. We further demonstrate its
versatility by extending layout guidance to the task of editing the layout and
context of a given real image.

通过提示文本控制生成的图像在空间布局方面存在问题，因此我们提出了一种名为布局引导技术的新方法，该方法可以改进模型的交叉注意层，以实现所需的重建方向，并成功进行了定量和定性的实验验证与扩展。