Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
Conventional methods usually rely on the assumptions of multi-view video
inputs, known camera parameters, or static scenes, all of which are typically
absent under in-the-wild scenarios. In this paper, we relax all these
constraints and tackle a highly ambitious but practical task, which we termed
as AnyV4D: we assume only one monocular video is available without any camera
parameters as input, and we aim to recover the dynamic 4D world alongside the
camera poses. To this end, we introduce GFlow, a new framework that utilizes
only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
representation, entailing a flow of Gaussian splatting through space and time.
GFlow first clusters the scene into still and moving parts, then applies a
sequential optimization process that optimizes camera poses and the dynamics of
3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
among neighboring points and smooth movement across frames. Since dynamic
scenes always introduce new content, we also propose a new pixel-wise
densification strategy for Gaussian points to integrate new visual content.
Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
enables tracking of any points across frames without the need for prior
training and segments moving objects from the scene in an unsupervised way.
Additionally, the camera poses of each frame can be derived from GFlow,
allowing for rendering novel views of a video scene through changing camera
pose. By employing the explicit representation, we may readily conduct
scene-level or object-level editing as desired, underscoring its versatility
and power. Visit our project website at: this https URL

使用 GFlow 框架，我们从单目视频中恢复动态的 4D 世界和相机姿态，通过使用深度和光流的 2D 先验进行聚类、优化和像素密集化，使得 GFlow 不仅可以进行 4D 重建，还可以进行对象追踪和无监督场景编辑。