Dynamic human rendering from video sequences has achieved remarkable progress
by formulating the rendering as a mapping from static poses to human images.
However, existing methods focus on the human appearance reconstruction of every
single frame while the temporal motion relations are not fully explored. In
this paper, we propose a new 4D motion modeling paradigm, SurMo, that jointly
models the temporal dynamics and human appearances in a unified framework with
three key designs: 1) Surface-based motion encoding that models 4D human
motions with an efficient compact surface-based triplane. It encodes both
spatial and temporal motion relations on the dense surface manifold of a
statistical body template, which inherits body topology priors for
generalizable novel view synthesis with sparse training observations. 2)
Physical motion decoding that is designed to encourage physical motion learning
by decoding the motion triplane features at timestep t to predict both spatial
derivatives and temporal derivatives at the next timestep t+1 in the training
stage. 3) 4D appearance decoding that renders the motion triplanes into images
by an efficient volumetric surface-conditioned renderer that focuses on the
rendering of body surfaces with motion learning conditioning. Extensive
experiments validate the state-of-the-art performance of our new paradigm and
illustrate the expressiveness of surface-based motion triplanes for rendering
high-fidelity view-consistent humans with fast motions and even
motion-dependent shadows. Our project page is at:
this https URL

提出了一种新的 4D 运动建模范例 SurMo，通过三个关键设计（基于表面的运动编码、物理运动解码和 4D 外观解码）在统一框架中联合建模时间动态和人体外貌。通过广泛实验证明了这种新范例的最先进性能，并展示了基于表面的运动三面体在渲染高保真度的一致视角人体时的表现能力。