We present latentSplat, a method to predict semantic Gaussians in a 3D latent space that can be splatted and decoded by a light-weight generative 2D architecture. Existing methods for generalizable 3D reconstruction either do not enable fast inference of high resolution novel views due to slow volume rendering, or are limited to interpolation of close input views, even in simpler settings with a single central object, where 360-degree generalization is possible. In this work, we combine a regression-based approach with a generative model, moving towards both of these capabilities within the same method, trained purely on readily available real video data. The core of our method are variational 3D Gaussians, a representation that efficiently encodes varying uncertainty within a latent space consisting of 3D feature Gaussians. From these Gaussians, specific instances can be sampled and rendered via efficient Gaussian splatting and a fast, generative decoder network. We show that latentSplat outperforms previous works in reconstruction quality and generalization, while being fast and scalable to high-resolution data.

我们提出了latentSplat，一种在3D潜在空间中预测语义高斯分布并能够通过轻量级生成的2D架构进行解码的方法。我们的方法结合了基于回归的方法和生成模型，通过训练纯粹基于现有真实视频数据，在相同的方法中实现了高分辨率新视图的快速推断和360度泛化插值。latentSplat在重建质量和泛化能力上优于先前的方法，并且对于高分辨率数据具有快速且可扩展的特性。

latentSplat：自动编码可变高斯用于快速可推广的3D重建