This paper proposes a neural radiance field (NeRF) approach for novel view
synthesis of dynamic scenes using forward warping. Existing methods often adopt
a static NeRF to represent the canonical space, and render dynamic images at
other time steps by mapping the sampled 3D points back to the canonical space
with the learned backward flow field. However, this backward flow field is
non-smooth and discontinuous, which is difficult to be fitted by commonly used
smooth motion models. To address this problem, we propose to estimate the
forward flow field and directly warp the canonical radiance field to other time
steps. Such forward flow field is smooth and continuous within the object
region, which benefits the motion model learning. To achieve this goal, we
represent the canonical radiance field with voxel grids to enable efficient
forward warping, and propose a differentiable warping process, including an
average splatting operation and an inpaint network, to resolve the many-to-one
and one-to-many mapping issues. Thorough experiments show that our method
outperforms existing methods in both novel view rendering and motion modeling,
demonstrating the effectiveness of our forward flow motion modeling. Project
page: this https URL

本研究提出了一种使用前向变形的神经辐射场 (NeRF) 方法，用于动态场景的新视角合成。通过在规范空间中表示静态 NeRF，并使用学习到的反向流场将采样的 3D 点映射回规范空间以渲染其他时间步骤的动态图像。然而，这种反向流场是非平滑和不连续的，难以适应常用的平滑运动模型。为解决这个问题，我们提出估计前向流场，并直接将规范辐射场变形到其他时间步骤。这种前向流场在对象区域内平滑和连续，有助于运动模型的学习。为实现这一目标，我们使用体素网格表示规范辐射场以实现高效的前向变形，并提出了可微分的变形过程，包括平均喷涂操作和修复网络，以解决多对一和一对多映射问题。详细实验证明，我们的方法在新视角合成和运动建模方面优于现有方法，验证了我们前向流动运动建模的有效性。

动态场景新视角合成的前向流

Forward Flow for Novel View Synthesis of Dynamic Scenes

Differentiable image sampling in the form of backward warping has seen broad
adoption in tasks like depth estimation and optical flow prediction. In
contrast, how to perform forward warping has seen less attention, partly due to
additional challenges such as resolving the conflict of mapping multiple pixels
to the same target location in a differentiable way. We propose softmax
splatting to address this paradigm shift and show its effectiveness on the
application of frame interpolation. Specifically, given two input frames, we
forward-warp the frames and their feature pyramid representations based on an
optical flow estimate using softmax splatting. In doing so, the softmax
splatting seamlessly handles cases where multiple source pixels map to the same
target location. We then use a synthesis network to predict the interpolation
result from the warped representations. Our softmax splatting allows us to not
only interpolate frames at an arbitrary time but also to fine tune the feature
pyramid and the optical flow. We show that our synthesis approach, empowered by
softmax splatting, achieves new state-of-the-art results for video frame
interpolation.

该论文提出了一种称为 softmax splatting 的方法，用于在前向映射过程中实现像素到目标位置的可微分映射，以及其在视频帧插值中的应用。