Text-driven 3D scene generation techniques have made rapid progress in recent
years. Their success is mainly attributed to using existing generative models
to iteratively perform image warping and inpainting to generate 3D scenes.
However, these methods heavily rely on the outputs of existing models, leading
to error accumulation in geometry and appearance that prevent the models from
being used in various scenarios (e.g., outdoor and unreal scenarios). To
address this limitation, we generatively refine the newly generated local views
by querying and aggregating global 3D information, and then progressively
generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF
as a unified representation of the 3D scene to constrain global 3D consistency,
and propose a generative refinement network to synthesize new contents with
higher quality by exploiting the natural image prior from 2D diffusion model as
well as the global 3D information of the current scene. Our extensive
experiments demonstrate that, in comparison to previous methods, our approach
supports wide variety of scene generation and arbitrary camera trajectories
with improved visual quality and 3D consistency.

利用文本驱动的 3D 场景生成技术，在构建 3D 场景时通过使用现有的生成模型进行图像变形和修复，同时结合查询和聚合全局 3D 信息来生成高质量的新内容，并在支持多种场景生成和任意相机路径的同时改善视觉效果和 3D 一致性。

3D-SceneDreamer: 文本驱动的 3D 一致场景生成

3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation

We introduce Text2Immersion, an elegant method for producing high-quality 3D
immersive scenes from text prompts. Our proposed pipeline initiates by
progressively generating a Gaussian cloud using pre-trained 2D diffusion and
depth estimation models. This is followed by a refining stage on the Gaussian
cloud, interpolating and refining it to enhance the details of the generated
scene. Distinct from prevalent methods that focus on single object or indoor
scenes, or employ zoom-out trajectories, our approach generates diverse scenes
with various objects, even extending to the creation of imaginary scenes.
Consequently, Text2Immersion can have wide-ranging implications for various
applications such as virtual reality, game development, and automated content
creation. Extensive evaluations demonstrate that our system surpasses other
methods in rendering quality and diversity, further progressing towards
text-driven 3D scene generation. We will make the source code publicly
accessible at the project page.

Text2Immersion 是一种从文本提示生成高质量 3D 沉浸式场景的优雅方法，可以生成多样化的场景，进一步推动了基于文本的 3D 场景生成。

Text2Immersion: 带有 3D 高斯的生成沉浸式场景

Text2Immersion: Generative Immersive Scene with 3D Gaussians

Text-driven 3D scene generation is widely applicable to video gaming, film
industry, and metaverse applications that have a large demand for 3D scenes.
However, existing text-to-3D generation methods are limited to producing 3D
objects with simple geometries and dreamlike styles that lack realism. In this
work, we present Text2NeRF, which is able to generate a wide range of 3D scenes
with complicated geometric structures and high-fidelity textures purely from a
text prompt. To this end, we adopt NeRF as the 3D representation and leverage a
pre-trained text-to-image diffusion model to constrain the 3D reconstruction of
the NeRF to reflect the scene description. Specifically, we employ the
diffusion model to infer the text-related image as the content prior and use a
monocular depth estimation method to offer the geometric prior. Both content
and geometric priors are utilized to update the NeRF model. To guarantee
textured and geometric consistency between different views, we introduce a
progressive scene inpainting and updating strategy for novel view synthesis of
the scene. Our method requires no additional training data but only a natural
language description of the scene as the input. Extensive experiments
demonstrate that our Text2NeRF outperforms existing methods in producing
photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of
natural language prompts.

Text2NeRF 是一种基于 NeRF 模型以自然语言描述为输入的 3D 场景生成方法，利用预训练的文本到图像扩散模型和单目深度估计方法约束 NeRF 模型以保持内容和几何一致，同时使用逐步场景修复和更新策略保证场景视角一致性。实验结果表明该方法能够生成高保真度，多视角一致性和多样性的真实感 3D 场景。