The blooming of virtual reality and augmented reality (VR/AR) technologies
has driven an increasing demand for the creation of high-quality, immersive,
and dynamic environments. However, existing generative techniques either focus
solely on dynamic objects or perform outpainting from a single perspective
image, failing to meet the needs of VR/AR applications. In this work, we tackle
the challenging task of elevating a single panorama to an immersive 4D
experience. For the first time, we demonstrate the capability to generate
omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
providing an immersive user experience. Our method introduces a pipeline that
facilitates natural scene animations and optimizes a set of 4D Gaussians using
efficient splatting techniques for real-time exploration. To overcome the lack
of scene-scale annotated 4D data and models, especially in panoramic formats,
we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
to animate consistently in 360-degree images, transforming them into panoramic
videos with dynamic scenes at targeted regions. Subsequently, we elevate the
panoramic video into a 4D immersive environment while preserving spatial and
temporal consistency. By transferring prior knowledge from 2D models in the
perspective domain to the panoramic domain and the 4D lifting with spatial
appearance and geometry regularization, we achieve high-quality Panorama-to-4D
generation at a resolution of (4096 $\times$ 2048) for the first time. See the
project website at this https URL

通过引入一种新的管道和技术，我们实现了将单个全景提升为沉浸式 4D 体验，生成 360 度视角下具有 4K 分辨率的全方位动态场景，并提供沉浸式用户体验。