Virtual 3D try-on can provide an intuitive and realistic view for online
shopping and has a huge potential commercial value. However, existing 3D
virtual try-on methods mainly rely on annotated 3D human shapes and garment
templates, which hinders their applications in practical scenarios. 2D virtual
try-on approaches provide a faster alternative to manipulate clothed humans,
but lack the rich and realistic 3D representation. In this paper, we propose a
novel Monocular-to-3D Virtual Try-On Network (M3D-VTON) that builds on the
merits of both 2D and 3D approaches. By integrating 2D information efficiently
and learning a mapping that lifts the 2D representation to 3D, we make the
first attempt to reconstruct a 3D try-on mesh only taking the target clothing
and a person image as inputs. The proposed M3D-VTON includes three modules: 1)
The Monocular Prediction Module (MPM) that estimates an initial full-body depth
map and accomplishes 2D clothes-person alignment through a novel two-stage
warping procedure; 2) The Depth Refinement Module (DRM) that refines the
initial body depth to produce more detailed pleat and face characteristics; 3)
The Texture Fusion Module (TFM) that fuses the warped clothing with the
non-target body part to refine the results. We also construct a high-quality
synthesized Monocular-to-3D virtual try-on dataset, in which each person image
is associated with a front and a back depth map. Extensive experiments
demonstrate that the proposed M3D-VTON can manipulate and reconstruct the 3D
human body wearing the given clothing with compelling details and is more
efficient than other 3D approaches.

该研究提出一种新颖的基于 2D 和 3D 方法的单目 - 3D 虚拟试穿网络，通过有效地整合 2D 信息和学习将 2D 表示提升到 3D 的映射以构建一个 3D 试穿模型。所提出的模型包含三个模块：单目预测模块、深度细化模块和纹理融合模块。该模型在构建高质量的试衣数据集的基础上，通过大量实验，证明了其比其他 3D 方法更高效和能够更真实地描述试穿效果。

M3D-VTON: 单目至 3D 虚拟试穿网络

M3D-VTON: A Monocular-to-3D Virtual Try-On Network

The 2D virtual try-on task has recently attracted a great interest from the
research community, for its direct potential applications in online shopping as
well as for its inherent and non-addressed scientific challenges. This task
requires fitting an in-shop cloth image on the image of a person, which is
highly challenging because it involves cloth warping, image compositing, and
synthesizing. Casting virtual try-on into a supervised task faces a difficulty:
available datasets are composed of pairs of pictures (cloth, person wearing the
cloth). Thus, we have no access to ground-truth when the cloth on the person
changes. State-of-the-art models solve this by masking the cloth information on
the person with both a human parser and a pose estimator. Then, image synthesis
modules are trained to reconstruct the person image from the masked person
image and the cloth image. This procedure has several caveats: firstly, human
parsers are prone to errors; secondly, it is a costly pre-processing step,
which also has to be applied at inference time; finally, it makes the task
harder than it is since the mask covers information that should be kept such as
hands or accessories. In this paper, we propose a novel student-teacher
paradigm where the teacher is trained in the standard way (reconstruction)
before guiding the student to focus on the initial task (changing the cloth).
The student additionally learns from an adversarial loss, which pushes it to
follow the distribution of the real images. Consequently, the student exploits
information that is masked to the teacher. A student trained without the
adversarial loss would not use this information. Also, getting rid of both
human parser and pose estimator at inference time allows obtaining a real-time
virtual try-on.

本文提出一种新的学生 - 教师模型来解决 2D 虚拟试穿任务中的监督学习问题，并且减少了图像处理的开销，从而实现了实时虚拟试穿。