We present latentSplat, a method to predict semantic Gaussians in a 3D latent
space that can be splatted and decoded by a light-weight generative 2D
architecture. Existing methods for generalizable 3D reconstruction either do
not enable fast inference of high resolution novel views due to slow volume
rendering, or are limited to interpolation of close input views, even in
simpler settings with a single central object, where 360-degree generalization
is possible. In this work, we combine a regression-based approach with a
generative model, moving towards both of these capabilities within the same
method, trained purely on readily available real video data. The core of our
method are variational 3D Gaussians, a representation that efficiently encodes
varying uncertainty within a latent space consisting of 3D feature Gaussians.
From these Gaussians, specific instances can be sampled and rendered via
efficient Gaussian splatting and a fast, generative decoder network. We show
that latentSplat outperforms previous works in reconstruction quality and
generalization, while being fast and scalable to high-resolution data.

我们提出了 latentSplat，一种在 3D 潜在空间中预测语义高斯分布并能够通过轻量级生成的 2D 架构进行解码的方法。我们的方法结合了基于回归的方法和生成模型，通过训练纯粹基于现有真实视频数据，在相同的方法中实现了高分辨率新视图的快速推断和 360 度泛化插值。latentSplat 在重建质量和泛化能力上优于先前的方法，并且对于高分辨率数据具有快速且可扩展的特性。