We introduce the first approach to solve the challenging problem of
unsupervised 4D visual scene understanding for complex dynamic scenes with
multiple interacting people from multi-view video. Our approach simultaneously
estimates a detailed model that includes a per-pixel semantically and
temporally coherent reconstruction, together with instance-level segmentation
exploiting photo-consistency, semantic and motion information. We further
leverage recent advances in 3D pose estimation to constrain the joint semantic
instance segmentation and 4D temporally coherent reconstruction. This enables
per person semantic instance segmentation of multiple interacting people in
complex dynamic scenes. Extensive evaluation of the joint visual scene
understanding framework against state-of-the-art methods on challenging indoor
and outdoor sequences demonstrates a significant (approx 40%) improvement in
semantic segmentation, reconstruction and scene flow accuracy.

这篇论文提出了使用多视角视频进行无监督 4D 视觉场景理解的方法，可以同时估计每个像素的具体模型和实例级别分割，并利用最近的 3D 姿势估计技术约束联合语义实例分割和 4D 时间上连贯的重建，以能够精准地分割多个动态场景中相互作用的人，实验表明与现有方法相比，语义分割、重建和场景流的准确性都显著提高了大约 40%。