In this work, we pioneer Semantic Flow, a neural semantic representation of
dynamic scenes from monocular videos. In contrast to previous NeRF methods that
reconstruct dynamic scenes from the colors and volume densities of individual
points, Semantic Flow learns semantics from continuous flows that contain rich
3D motion information. As there is 2D-to-3D ambiguity problem in the viewing
direction when extracting 3D flow features from 2D video frames, we consider
the volume densities as opacity priors that describe the contributions of flow
features to the semantics on the frames. More specifically, we first learn a
flow network to predict flows in the dynamic scene, and propose a flow feature
aggregation module to extract flow features from video frames. Then, we propose
a flow attention module to extract motion information from flow features, which
is followed by a semantic network to output semantic logits of flows. We
integrate the logits with volume densities in the viewing direction to
supervise the flow features with semantic labels on video frames. Experimental
results show that our model is able to learn from multiple dynamic scenes and
supports a series of new tasks such as instance-level scene editing, semantic
completions, dynamic scene tracking and semantic adaption on novel scenes.
Codes are available at this https URL

本文中，我们提出了 Semantic Flow，一种从单 ocular videos 中学习动态场景的神经语义表示。与以往的 NeRF 方法不同，它通过包含丰富的三维运动信息的连续流来学习语义。我们考虑从二维视频帧中提取三维流特征时视线方向上的二维到三维模糊问题，将体积密度视为描述流特征对帧语义贡献的不透明度先验。我们首先学习流网络预测动态场景中的流，并提出流特征聚合模块从视频帧中提取流特征。然后，我们提出流注意模块从流特征中提取运动信息，随后是语义网络输出流的语义 logits。我们将 logits 与视线方向上的体积密度相结合，以在视频帧上监督流特征与语义标签的学习。实验结果表明，我们的模型能够学习多个动态场景，并支持一系列新任务，如实例级场景编辑、语义补全、动态场景跟踪和对新场景的语义适应。可在此 https URL 获取代码。