Virtual try-on has become a popular research topic, but most existing methods
focus on studio images with a clean background. They can achieve plausible
results for this studio try-on setting by learning to warp a garment image to
fit a person's body from paired training data, i.e., garment images paired with
images of people wearing the same garment. Such data is often collected from
commercial websites, where each garment is demonstrated both by itself and on
several models. By contrast, it is hard to collect paired data for in-the-wild
scenes, and therefore, virtual try-on for casual images of people against
cluttered backgrounds is rarely studied.
In this work, we fill the gap in the current virtual try-on research by (1)
introducing a Street TryOn benchmark to evaluate performance on street scenes
and (2) proposing a novel method that can learn without paired data, from a set
of in-the-wild person images directly. Our method can achieve robust
performance across shop and street domains using a novel DensePose warping
correction method combined with diffusion-based inpainting controlled by pose
and semantic segmentation. Our experiments demonstrate competitive performance
for standard studio try-on tasks and SOTA performance for street try-on and
cross-domain try-on tasks.

在这项工作中，我们通过（1）引入一个街景尝试基准来评估在街景中的性能，以及（2）提出一种可以直接从一组野外人物图像中学习而不需要成对数据的新方法，来填补当前虚拟试穿研究中的空白。我们的方法可以通过结合基于 DensePose 的变形校正方法和由姿势和语义分割控制的扩散式修补方法，在商店和街道领域实现稳健的性能，并证明在标准工作室试穿任务和街头试穿和跨领域试穿任务方面具有 SOTA 性能。

街头试衣：从非配对人物图像中学习自然环境下的虚拟试衣

Street TryOn: Learning In-the-Wild Virtual Try-On from Unpaired Person  Images

We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
synthesis for in-the-wild scenes. While existing methods are designed for
single objects with masked backgrounds, we propose new techniques to address
challenges introduced by in-the-wild multi-object scenes with complex
backgrounds. Specifically, we train a generative prior on a mixture of data
sources that capture object-centric, indoor, and outdoor scenes. To address
issues from data mixture such as depth-scale ambiguity, we propose a novel
camera conditioning parameterization and normalization scheme. Further, we
observe that Score Distillation Sampling (SDS) tends to truncate the
distribution of complex backgrounds during distillation of 360-degree scenes,
and propose "SDS anchoring" to improve the diversity of synthesized novel
views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
in the zero-shot setting, even outperforming methods specifically trained on
DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
for single-image novel view synthesis, and demonstrate strong performance in
this setting. Our code and data are at this http URL

我们引入了一个三维感知扩散模型 ZeroNVS，用于野外场景下的单图像新视图合成。通过训练一种生成式先验模型来处理多物体场景和复杂背景带来的挑战，提出了新的技术。我们还提出了一种新颖的相机条件参数化和归一化方案，以解决深度尺度的二义性问题。此外，我们注意到 Score Distillation Sampling（SDS）在蒸馏 360 度场景时倾向于截断复杂背景的分布，并提出了 “SDS anchoring” 以改善合成新视图的多样性。我们的模型在 DTU 数据集的零样本设置中取得了新的 LPIPS 优势，甚至优于专门在 DTU 上训练的方法。我们进一步将具有挑战性的 Mip-NeRF 360 数据集调整为单图像新视图合成的新基准，并在该设置中展现出强大的性能。我们的代码和数据位于此 http URL。

ZeroNVS: 从单张真实图像实现零射击 360 度视角综合

ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image

Convolutional Neural Network based approaches for monocular 3D human pose
estimation usually require a large amount of training images with 3D pose
annotations. While it is feasible to provide 2D joint annotations for large
corpora of in-the-wild images with humans, providing accurate 3D annotations to
such in-the-wild corpora is hardly feasible in practice. Most existing 3D
labelled data sets are either synthetically created or feature in-studio
images. 3D pose estimation algorithms trained on such data often have limited
ability to generalize to real world scene diversity. We therefore propose a new
deep learning based method for monocular 3D human pose estimation that shows
high accuracy and generalizes better to in-the-wild scenes. It has a network
architecture that comprises a new disentangled hidden space encoding of
explicit 2D and 3D features, and uses supervision by a new learned projection
model from predicted 3D pose. Our algorithm can be jointly trained on image
data with 3D labels and image data with only 2D labels. It achieves
state-of-the-art accuracy on challenging in-the-wild data.

本文提出了一种基于卷积神经网络的深度学习方法，用于单目 3D 人体姿态估计，具有高精度和更好的野外场景泛化能力，可以联合在具有 3D 标签和仅有 2D 标签的图像数据上进行训练，并在具有挑战性的野外数据上实现了最先进的准确性。