This paper presents GGRt, a novel approach to generalizable novel view
synthesis that alleviates the need for real camera poses, complexity in
processing high-resolution images, and lengthy optimization processes, thus
facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in
real-world scenarios. Specifically, we design a novel joint learning framework
that consists of an Iterative Pose Optimization Network (IPO-Net) and a
Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism,
the proposed framework can inherently estimate robust relative pose information
from the image observations and thus primarily alleviate the requirement of
real camera poses. Moreover, we implement a deferred back-propagation mechanism
that enables high-resolution training and inference, overcoming the resolution
constraints of previous methods. To enhance the speed and efficiency, we
further introduce a progressive Gaussian cache module that dynamically adjusts
during training and inference. As the first pose-free generalizable 3D-GS
framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at
$\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our
method outperforms existing NeRF-based pose-free techniques in terms of
inference speed and effectiveness. It can also approach the real pose-based
3D-GS methods. Our contributions provide a significant leap forward for the
integration of computer vision and computer graphics into practical
applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open
datasets and enabling real-time rendering for immersive experiences.

这篇论文介绍了 GGRt，一种新颖的通用新视角合成方法，它减轻了对真实相机位姿、处理高分辨率图像的复杂性和冗长的优化过程的需要，从而促进了 3D 高斯喷溅（3D-GS）在实际场景中的更强适用性。