Neural Radiance Fields (NeRF) revolutionize the realm of visual media by
providing photorealistic Free-Viewpoint Video (FVV) experiences, offering
viewers unparalleled immersion and interactivity. However, the technology's
significant storage requirements and the computational complexity involved in
generation and rendering currently limit its broader application. To close this
gap, this paper presents Temporal Tri-Plane Radiance Fields (TeTriRF), a novel
technology that significantly reduces the storage size for Free-Viewpoint Video
(FVV) while maintaining low-cost generation and rendering. TeTriRF introduces a
hybrid representation with tri-planes and voxel grids to support scaling up to
long-duration sequences and scenes with complex motions or rapid changes. We
propose a group training scheme tailored to achieving high training efficiency
and yielding temporally consistent, low-entropy scene representations.
Leveraging these properties of the representations, we introduce a compression
pipeline with off-the-shelf video codecs, achieving an order of magnitude less
storage size compared to the state-of-the-art. Our experiments demonstrate that
TeTriRF can achieve competitive quality with a higher compression rate.

本研究提出了 Temporoal Tri-Plane Radiance Fields（TeTriRF），通过引入三平面和体素网格的混合表示方法，支持长时序列和场景的大规模扩展，并使用现成的视频编解码器，实现与现有技术相比一个数量级的存储空间减小，同时保持高压缩率和竞争性的质量。

TeTriRF：用于高效自由视角视频的时域三平面辐射场

TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint  Video

In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary
human performance using sparse multi-view cameras. Recently, several works have
addressed this problem by learning person-specific neural radiance fields
(NeRF) to capture the appearance of a particular human. In parallel, some work
proposed to use pixel-aligned features to generalize radiance fields to
arbitrary new scenes and objects. Adopting such generalization approaches to
humans, however, is highly challenging due to the heavy occlusions and dynamic
articulations of body parts. To tackle this, we propose Neural Human Performer,
a novel approach that learns generalizable neural radiance fields based on a
parametric human body model for robust performance capture. Specifically, we
first introduce a temporal transformer that aggregates tracked visual features
based on the skeletal body motion over time. Moreover, a multi-view transformer
is proposed to perform cross-attention between the temporally-fused features
and the pixel-aligned features at each time step to integrate observations on
the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets
show that our method significantly outperforms recent generalizable NeRF
methods on unseen identities and poses. The video results and code are
available at this https URL

本研究提出了一种基于人体参数模型学习可泛化神经辐射场的新方法，使用稀疏多视角相机合成任意人类表演的自由视点视频，并在实验中取得了明显的优于以往 NeRF 方法的表现。

神经人类表现者：学习通用辐射场以进行人类表现渲染

Neural Human Performer: Learning Generalizable Radiance Fields for Human  Performance Rendering

Generating free-viewpoint videos is critical for immersive VR/AR experience
but recent neural advances still lack the editing ability to manipulate the
visual perception for large dynamic scenes. To fill this gap, in this paper we
propose the first approach for editable photo-realistic free-viewpoint video
generation for large-scale dynamic scenes using only sparse 16 cameras. The
core of our approach is a new layered neural representation, where each dynamic
entity including the environment itself is formulated into a space-time
coherent neural layered radiance representation called ST-NeRF. Such layered
representation supports fully perception and realistic manipulation of the
dynamic scene whilst still supporting a free viewing experience in a wide
range. In our ST-NeRF, the dynamic entity/layer is represented as continuous
functions, which achieves the disentanglement of location, deformation as well
as the appearance of the dynamic entity in a continuous and self-supervised
manner. We propose a scene parsing 4D label map tracking to disentangle the
spatial information explicitly, and a continuous deform module to disentangle
the temporal motion implicitly. An object-aware volume rendering scheme is
further introduced for the re-assembling of all the neural layers. We adopt a
novel layered loss and motion-aware ray sampling strategy to enable efficient
training for a large dynamic scene with multiple performers, Our framework
further enables a variety of editing functions, i.e., manipulating the scale
and location, duplicating or retiming individual neural layers to create
numerous visual effects while preserving high realism. Extensive experiments
demonstrate the effectiveness of our approach to achieve high-quality,
photo-realistic, and editable free-viewpoint video generation for dynamic
scenes.

本文提出了一种可编辑的逼真的自由视角视频生成方法，通过使用只有 16 个稀疏摄像头，将大规模动态场景中的每个动态实体表示为连续函数，并采用新的分层神经表示方法进行支持感知和逼真操作，并提出使用场景解析 4D 标签地图追踪和连续变形模块隐式分离时间运动，同时采用面向对象的体积渲染方案进行了再组装所有神经层，并采用新颖的分层损失和运动感知射线采样策略，实现了大规模动态场景的高效训练，并支持各种编辑功能，显示了该方法生成高质量的可编辑的自由视角视频。

使用分层神经表示的可编辑自由视角视频

Editable Free-viewpoint Video Using a Layered Neural Representation

The availability of affordable 3D full body reconstruction systems has given
rise to free-viewpoint video (FVV) of human shapes. Most existing solutions
produce temporally uncorrelated point clouds or meshes with unknown
point/vertex correspondences. Individually compressing each frame is
ineffective and still yields to ultra-large data sizes. We present an
end-to-end deep learning scheme to establish dense shape correspondences and
subsequently compress the data. Our approach uses sparse set of "panoramic"
depth maps or PDMs, each emulating an inward-viewing concentric mosaics. We
then develop a learning-based technique to learn pixel-wise feature descriptors
on PDMs. The results are fed into an autoencoder-based network for compression.
Comprehensive experiments demonstrate our solution is robust and effective on
both public and our newly captured datasets.

采用端到端的深度学习方法建立密集的形状对应关系，并随后压缩 3D 全身重建数据。