We present Monocular Neural Parametric Head Models (MonoNPHM) for dynamic 3D
head reconstructions from monocular RGB videos. To this end, we propose a
latent appearance space that parameterizes a texture field on top of a neural
parametric model. We constrain predicted color values to be correlated with the
underlying geometry such that gradients from RGB effectively influence latent
geometry codes during inverse rendering. To increase the representational
capacity of our expression space, we augment our backward deformation field
with hyper-dimensions, thus improving color and geometry representation in
topologically challenging expressions. Using MonoNPHM as a learned prior, we
approach the task of 3D head reconstruction using signed distance field based
volumetric rendering. By numerically inverting our backward deformation field,
we incorporated a landmark loss using facial anchor points that are closely
tied to our canonical geometry representation. To evaluate the task of dynamic
face reconstruction from monocular RGB videos we record 20 challenging Kinect
sequences under casual conditions. MonoNPHM outperforms all baselines with a
significant margin, and makes an important step towards easily accessible
neural parametric face models through RGB tracking.

动态的 3D 头部重建，利用单目 RGB 视频进行 Monocular Neural Parametric Head Models (MonoNPHM) 的建模。

MonoNPHM：从单目视频中动态重建头部

MonoNPHM: Dynamic Head Reconstruction from Monocular Videos

Existing methods for 3D tracking from monocular RGB videos predominantly
consider articulated and rigid objects. Modelling dense non-rigid object
deformations in this setting remained largely unaddressed so far, although such
effects can improve the realism of the downstream applications such as AR/VR
and avatar communications. This is due to the severe ill-posedness of the
monocular view setting and the associated challenges. While it is possible to
naively track multiple non-rigid objects independently using 3D templates or
parametric 3D models, such an approach would suffer from multiple artefacts in
the resulting 3D estimates such as depth ambiguity, unnatural intra-object
collisions and missing or implausible deformations. Hence, this paper
introduces the first method that addresses the fundamental challenges depicted
above and that allows tracking human hands interacting with human faces in 3D
from single monocular RGB videos. We model hands as articulated objects
inducing non-rigid face deformations during an active interaction. Our method
relies on a new hand-face motion and interaction capture dataset with realistic
face deformations acquired with a markerless multi-view camera system. As a
pivotal step in its creation, we process the reconstructed raw 3D shapes with
position-based dynamics and an approach for non-uniform stiffness estimation of
the head tissues, which results in plausible annotations of the surface
deformations, hand-face contact regions and head-hand positions. At the core of
our neural approach are a variational auto-encoder supplying the hand-face
depth prior and modules that guide the 3D tracking by estimating the contacts
and the deformations. Our final 3D hand and face reconstructions are realistic
and more plausible compared to several baselines applicable in our setting,
both quantitatively and qualitatively.
this https URL

从单目 RGB 视频中追踪人手与人脸之间的 3D 交互，并建立了可靠的 3D 手部和面部重构方法。

Decaf：面部和手部交互的单眼变形捕捉

Decaf: Monocular Deformation Capture for Face and Hand Interactions

Human motion synthesis is an important problem with applications in graphics,
gaming and simulation environments for robotics. Existing methods require
accurate motion capture data for training, which is costly to obtain. Instead,
we propose a framework for training generative models of physically plausible
human motion directly from monocular RGB videos, which are much more widely
available. At the core of our method is a novel optimization formulation that
corrects imperfect image-based pose estimations by enforcing physics
constraints and reasons about contacts in a differentiable way. This
optimization yields corrected 3D poses and motions, as well as their
corresponding contact forces. Results show that our physically-corrected
motions significantly outperform prior work on pose estimation. We can then use
these to train a generative model to synthesize future motion. We demonstrate
both qualitatively and quantitatively improved motion estimation, synthesis
quality and physical plausibility achieved by our method on the Human3.6m
dataset~\cite{h36m_pami} as compared to prior kinematic and physics-based
methods. By enabling learning of motion synthesis from video, our method paves
the way for large-scale, realistic and diverse motion synthesis. Project page:
https://nv-tlabs.github.io/publication/iccv_2021_physics/

通过单眼 RGB 视频直接训练具有物理可行性的人体运动的生成模型，该方法通过可微分的方式实现了物理约束和接触关系优化，并实现了与之前基于姿态识别的方法相比更高质量、更真实、更多样化的运动合成与估计。

基于物理的视频人体运动估计和合成

Physics-based Human Motion Estimation and Synthesis from Videos

This paper proposes GraviCap, i.e., a new approach for joint markerless 3D
human motion capture and object trajectory estimation from monocular RGB
videos. We focus on scenes with objects partially observed during a free
flight. In contrast to existing monocular methods, we can recover scale, object
trajectories as well as human bone lengths in meters and the ground plane's
orientation, thanks to the awareness of the gravity constraining object
motions. Our objective function is parametrised by the object's initial
velocity and position, gravity direction and focal length, and jointly
optimised for one or several free flight episodes. The proposed human-object
interaction constraints ensure geometric consistency of the 3D reconstructions
and improved physical plausibility of human poses compared to the unconstrained
case. We evaluate GraviCap on a new dataset with ground-truth annotations for
persons and different objects undergoing free flights. In the experiments, our
approach achieves state-of-the-art accuracy in 3D human motion capture on
various metrics. We urge the reader to watch our supplementary video. Both the
source code and the dataset are released; see
this http URL

本文提出 GraviCap，即一种新的联合无标记 3D 人体动作捕捉和单目 RGB 视频对象轨迹估计的方法。该方法利用重力约束物体运动，可以恢复具有部分观测对象的场景的尺度、对象轨迹、人体骨长和地平面方向。通过添加人对象交互约束，可以提高 3D 重建的几何一致性和改善人体姿势的物理合理性。在一系列新数据集上测试表明，该方法在 3D 人体动作捕捉上具有最先进的准确性。