We propose a novel representation of virtual humans for highly realistic
real-time animation and rendering in 3D applications. We learn pose dependent
appearance and geometry from highly accurate dynamic mesh sequences obtained
from state-of-the-art multiview-video reconstruction. Learning pose-dependent
appearance and geometry from mesh sequences poses significant challenges, as it
requires the network to learn the intricate shape and articulated motion of a
human body. However, statistical body models like SMPL provide valuable
a-priori knowledge which we leverage in order to constrain the dimension of the
search space enabling more efficient and targeted learning and define
pose-dependency. Instead of directly learning absolute pose-dependent geometry,
we learn the difference between the observed geometry and the fitted SMPL
model. This allows us to encode both pose-dependent appearance and geometry in
the consistent UV space of the SMPL model. This approach not only ensures a
high level of realism but also facilitates streamlined processing and rendering
of virtual humans in real-time scenarios.

我们提出了一种新颖的虚拟人物表示方法，用于高度逼真的实时动画和渲染 3D 应用程序。我们从最先进的多视角视频重建中获取高精度的动态网格序列，学习姿态相关的外观和几何形状，并利用统计身体模型来限制搜索空间，以实现更高效、有针对性的学习和定义姿态依赖性。通过在 SMPL 模型的一致 UV 空间中学习观察到的几何与拟合的 SMPL 模型之间的差异，我们能够在模型中编码姿态相关的外观和几何形状，从而不仅保证了较高水平的真实感，还促进了虚拟人物的实时处理和渲染。