We present M&M VTO, a mix and match virtual try-on method that takes as input
multiple garment images, text description for garment layout and an image of a
person. An example input includes: an image of a shirt, an image of a pair of
pants, "rolled sleeves, shirt tucked in", and an image of a person. The output
is a visualization of how those garments (in the desired layout) would look
like on the given person. Key contributions of our method are: 1) a single
stage diffusion based model, with no super resolution cascading, that allows to
mix and match multiple garments at 1024x512 resolution preserving and warping
intricate garment details, 2) architecture design (VTO UNet Diffusion
Transformer) to disentangle denoising from person specific features, allowing
for a highly effective finetuning strategy for identity preservation (6MB model
per individual vs 4GB achieved with, e.g., dreambooth finetuning); solving a
common identity loss problem in current virtual try-on methods, 3) layout
control for multiple garments via text inputs specifically finetuned over
PaLI-3 for virtual try-on task. Experimental results indicate that M&M VTO
achieves state-of-the-art performance both qualitatively and quantitatively, as
well as opens up new opportunities for virtual try-on via language-guided and
multi-garment try-on.

M&M VTO 是一种多样式虚拟试穿方法，以多个服装图像、服装布局的文本描述和人物图像作为输入，并输出展示给定人物上穿指定布局的服装效果的可视化结果。

M&M VTO: 多款服装虚拟试穿和编辑

M&M VTO: Multi-Garment Virtual Try-On and Editing

A virtual try-on method takes a product image and an image of a model and
produces an image of the model wearing the product. Most methods essentially
compute warps from the product image to the model image and combine using image
generation methods. However, obtaining a realistic image is challenging because
the kinematics of garments is complex and because outline, texture, and shading
cues in the image reveal errors to human viewers. The garment must have
appropriate drapes; texture must be warped to be consistent with the shape of a
draped garment; small details (buttons, collars, lapels, pockets, etc.) must be
placed appropriately on the garment, and so on. Evaluation is particularly
difficult and is usually qualitative.
This paper uses quantitative evaluation on a challenging, novel dataset to
demonstrate that (a) for any warping method, one can choose target models
automatically to improve results, and (b) learning multiple coordinated
specialized warpers offers further improvements on results. Target models are
chosen by a learned embedding procedure that predicts a representation of the
products the model is wearing. This prediction is used to match products to
models. Specialized warpers are trained by a method that encourages a second
warper to perform well in locations where the first works poorly. The warps are
then combined using a U-Net. Qualitative evaluation confirms that these
improvements are wholesale over outline, texture shading, and garment details.

本篇论文提出了一种深度学习的虚拟试穿方法，针对传统方法在衣物形态、质感等方面存在的问题进行了改善，并使用量化评估方法对模型的性能进行了验证。