The success of the Neural Radiance Fields (NeRFs) for modeling and free-view
rendering static objects has inspired numerous attempts on dynamic scenes.
Current techniques that utilize neural rendering for facilitating free-view
videos (FVVs) are restricted to either offline rendering or are capable of
processing only brief sequences with minimal motion. In this paper, we present
a novel technique, Residual Radiance Field or ReRF, as a highly compact neural
representation to achieve real-time FVV rendering on long-duration dynamic
scenes. ReRF explicitly models the residual information between adjacent
timestamps in the spatial-temporal feature space, with a global
coordinate-based tiny MLP as the feature decoder. Specifically, ReRF employs a
compact motion grid along with a residual feature grid to exploit inter-frame
feature similarities. We show such a strategy can handle large motions without
sacrificing quality. We further present a sequential training scheme to
maintain the smoothness and the sparsity of the motion/residual grids. Based on
ReRF, we design a special FVV codec that achieves three orders of magnitudes
compression rate and provides a companion ReRF player to support online
streaming of long-duration FVVs of dynamic scenes. Extensive experiments
demonstrate the effectiveness of ReRF for compactly representing dynamic
radiance fields, enabling an unprecedented free-viewpoint viewing experience in
speed and quality.

本文提出了一种名为 Residual Radiance Field (ReRF) 的新技术，用于高效压缩长时间动态场景的实时自由视点视频渲染，借助全局坐标小型 MLP 作为特征解码器。通过使用紧凑的运动网格以及残差特征网格来利用帧间特征相似性，我们表明这样的策略可以处理大运动而不降低质量，并且提供了一个特殊的 FVV 编解码器和一个 ReRF 播放器，支持在线流式传输长时间动态场景的 FVVs，并通过大量实验展示了 ReRF 在动态辐射场紧凑表示方面的有效性。