Modeling and rendering of dynamic scenes is challenging, as natural scenes
often contain complex phenomena such as thin structures, evolving topology,
translucency, scattering, occlusion, and biological motion. Mesh-based
reconstruction and tracking often fail in these cases, and other approaches
(e.g., light field video) typically rely on constrained viewing conditions,
which limit interactivity. We circumvent these difficulties by presenting a
learning-based approach to representing dynamic objects inspired by the
integral projection model used in tomographic imaging. The approach is
supervised directly from 2D images in a multi-view capture setting and does not
require explicit reconstruction or tracking of the object. Our method has two
primary components: an encoder-decoder network that transforms input images
into a 3D volume representation, and a differentiable ray-marching operation
that enables end-to-end training. By virtue of its 3D representation, our
construction extrapolates better to novel viewpoints compared to screen-space
rendering techniques. The encoder-decoder architecture learns a latent
representation of a dynamic scene that enables us to produce novel content
sequences not seen during training. To overcome memory limitations of
voxel-based representations, we learn a dynamic irregular grid structure
implemented with a warp field during ray-marching. This structure greatly
improves the apparent resolution and reduces grid-like artifacts and jagged
motion. Finally, we demonstrate how to incorporate surface-based
representations into our volumetric-learning framework for applications where
the highest resolution is required, using facial performance capture as a case
in point.

通过学习，我们提出了一种表示动态物体的方法，该方法由编码器 - 解码器网络和可微分的射线行进操作组成，它不需要明确重建或跟踪物体，使用了 3D 体积表示法，表现出更好的图像质量，为高分辨率应用提供了一种从表面到体积的方法