Video representation is a long-standing problem that is crucial for various
down-stream tasks, such as tracking,depth prediction,segmentation,view
synthesis,and editing. However, current methods either struggle to model
complex motions due to the absence of 3D structure or rely on implicit 3D
representations that are ill-suited for manipulation tasks. To address these
challenges, we introduce a novel explicit 3D representation-video Gaussian
representation -- that embeds a video into 3D Gaussians. Our proposed
representation models video appearance in a 3D canonical space using explicit
Gaussians as proxies and associates each Gaussian with 3D motions for video
motion. This approach offers a more intrinsic and explicit representation than
layered atlas or volumetric pixel matrices. To obtain such a representation, we
distill 2D priors, such as optical flow and depth, from foundation models to
regularize learning in this ill-posed setting. Extensive applications
demonstrate the versatility of our new video representation. It has been proven
effective in numerous video processing tasks, including tracking, consistent
video depth and feature refinement, motion and appearance editing, and
stereoscopic video generation. Project page:
this https URL

我们引入了一种新颖的显式三维表示 - 视频高斯表示，用三维高斯将视频嵌入其中，以模拟视频的外观和运动。该方法在多个视频处理任务中表现出了高效性。