Gaussian Splatting (GS) has significantly elevated scene reconstruction
efficiency and novel view synthesis (NVS) accuracy compared to Neural Radiance
Fields (NeRF), particularly for dynamic scenes. However, current 4D NVS
methods, whether based on GS or NeRF, primarily rely on camera parameters
provided by COLMAP and even utilize sparse point clouds generated by COLMAP for
initialization, which lack accuracy as well are time-consuming. This sometimes
results in poor dynamic scene representation, especially in scenes with large
object movements, or extreme camera conditions e.g. small translations combined
with large rotations. Some studies simultaneously optimize the estimation of
camera parameters and scenes, supervised by additional information like depth,
optical flow, etc. obtained from off-the-shelf models. Using this unverified
information as ground truth can reduce robustness and accuracy, which does
frequently occur for long monocular videos (with e.g. > hundreds of frames). We
propose a novel approach that learns a high-fidelity 4D GS scene representation
with self-calibration of camera parameters. It includes the extraction of 2D
point features that robustly represent 3D structure, and their use for
subsequent joint optimization of camera parameters and 3D structure towards
overall 4D scene optimization. We demonstrate the accuracy and time efficiency
of our method through extensive quantitative and qualitative experimental
results on several standard benchmarks. The results show significant
improvements over state-of-the-art methods for 4D novel view synthesis. The
source code will be released soon at this https URL

通过学习高保真度的 4D 高斯点云分割场景表征，自校准相机参数，以及联合优化相机参数和 3D 结构，我们的方法在 4D 新视图合成方面展现出了显著的改进。