We present MultiPly, a novel framework to reconstruct multiple people in 3D
from monocular in-the-wild videos. Reconstructing multiple individuals moving
and interacting naturally from monocular in-the-wild videos poses a challenging
task. Addressing it necessitates precise pixel-level disentanglement of
individuals without any prior knowledge about the subjects. Moreover, it
requires recovering intricate and complete 3D human shapes from short video
sequences, intensifying the level of difficulty. To tackle these challenges, we
first define a layered neural representation for the entire scene, composited
by individual human and background models. We learn the layered neural
representation from videos via our layer-wise differentiable volume rendering.
This learning process is further enhanced by our hybrid instance segmentation
approach which combines the self-supervised 3D segmentation and the promptable
2D segmentation module, yielding reliable instance segmentation supervision
even under close human interaction. A confidence-guided optimization
formulation is introduced to optimize the human poses and shape/appearance
alternately. We incorporate effective objectives to refine human poses via
photometric information and impose physically plausible constraints on human
dynamics, leading to temporally consistent 3D reconstructions with high
fidelity. The evaluation of our method shows the superiority over prior art on
publicly available datasets and in-the-wild videos.

通过我的研究，我们提出了一种名为 MultiPly 的新型框架，能够从野外单目视频中重建多个人的 3D 模型。该框架利用分层神经表示来还原整个场景，其中包括个体人物和背景模型，并通过层次不可分离的体积渲染学习这一分层神经表示。我们的混合实例分割方法结合了自我监督的 3D 分割和可提示的 2D 分割模块，可以在近距离人体交互的情况下得到可靠的实例分割辅助训练。我们引入了一种基于置信度的优化公式，交替优化人体姿势和形状 / 外貌。通过利用光度信息改善人体姿势，并对人体动态施加物理可行性约束，我们达到了具有高度保真度的时间一致的 3D 重建结果。我们的方法在公开数据集和野外视频上展现了超越先前研究成果的优越性能。

MultiPly: 野外单目视频中多人重建

MultiPly: Reconstruction of Multiple People from Monocular Video in the  Wild

Existing VLMs can track in-the-wild 2D video objects while current generative
models provide powerful visual priors for synthesizing novel views for the
highly under-constrained 2D-to-3D object lifting. Building upon this exciting
progress, we present DreamScene4D, the first approach that can generate
three-dimensional dynamic scenes of multiple objects from monocular in-the-wild
videos with large object motion across occlusions and novel viewpoints. Our key
insight is to design a "decompose-then-recompose" scheme to factorize both the
whole video scene and each object's 3D motion. We first decompose the video
scene by using open-vocabulary mask trackers and an adapted image diffusion
model to segment, track, and amodally complete the objects and background in
the video. Each object track is mapped to a set of 3D Gaussians that deform and
move in space and time. We also factorize the observed motion into multiple
components to handle fast motion. The camera motion can be inferred by
re-rendering the background to match the video frames. For the object motion,
we first model the object-centric deformation of the objects by leveraging
rendering losses and multi-view generative priors in an object-centric frame,
then optimize object-centric to world-frame transformations by comparing the
rendered outputs against the perceived pixel and optical flow. Finally, we
recompose the background and objects and optimize for relative object scales
using monocular depth prediction guidance. We show extensive results on the
challenging DAVIS, Kubric, and self-captured videos, detail some limitations,
and provide future directions. Besides 4D scene generation, our results show
that DreamScene4D enables accurate 2D point motion tracking by projecting the
inferred 3D trajectories to 2D, while never explicitly trained to do so.

DreamScene4D 是一种能够从野外单眼视频中生成多个物体的三维动态场景的方法，通过设计 “分解 - 重组” 方案，将视频场景及每个物体的三维运动进行分解，并使用开放词汇遮罩跟踪和适应性图像扩散模型对视频中的物体和背景进行分割、追踪和完整修复。