Neural radiance fields have made a remarkable breakthrough in the novel view
synthesis task at the 3D static scene. However, for the 4D circumstance (e.g.,
dynamic scene), the performance of the existing method is still limited by the
capacity of the neural network, typically in a multilayer perceptron network
(MLP). In this paper, we utilize 3D Voxel to model the 4D neural radiance
field, short as V4D, where the 3D voxel has two formats. The first one is to
regularly model the 3D space and then use the sampled local 3D feature with the
time index to model the density field and the texture field by a tiny MLP. The
second one is in look-up tables (LUTs) format that is for the pixel-level
refinement, where the pseudo-surface produced by the volume rendering is
utilized as the guidance information to learn a 2D pixel-level refinement
mapping. The proposed LUTs-based refinement module achieves the performance
gain with little computational cost and could serve as the plug-and-play module
in the novel view synthesis task. Moreover, we propose a more effective
conditional positional encoding toward the 4D data that achieves performance
gain with negligible computational burdens. Extensive experiments demonstrate
that the proposed method achieves state-of-the-art performance at a low
computational cost.

本文提出了一种利用 3D Voxel 建模 4D 神经辐射场的方法，即 V4D, 并使用 LUTs 格式和条件位置编码来改进新视图综合任务的表现，实验表明该方法在低计算成本下取得了最先进的表现。

V4D: 4D 视角合成的体素方法

V4D: Voxel for 4D Novel View Synthesis

Most existing 3D CNNs for video representation learning are clip-based
methods, and thus do not consider video-level temporal evolution of
spatio-temporal features. In this paper, we propose Video-level 4D
Convolutional Neural Networks, referred as V4D, to model the evolution of
long-range spatio-temporal representation with 4D convolutions, and at the same
time, to preserve strong 3D spatio-temporal representation with residual
connections. Specifically, we design a new 4D residual block able to capture
inter-clip interactions, which could enhance the representation power of the
original clip-level 3D CNNs. The 4D residual blocks can be easily integrated
into the existing 3D CNNs to perform long-range modeling hierarchically. We
further introduce the training and inference methods for the proposed V4D.
Extensive experiments are conducted on three video recognition benchmarks,
where V4D achieves excellent results, surpassing recent 3D CNNs by a large
margin.

本文提出采用 Video-level 4D 卷积神经网络（V4D）模型的长时空特征演化，通过设计新的 4D 残差块捕捉视频间的交互作用，将 4D 残差块轻松整合到现有的 3D CNN 中以进行层次化的长程建模，并在三个视频识别基准数据集上进行了广泛的实验验证，V4D 表现出优异的结果，显著优于最近的 3D CNN。