The generation of 3D scenes from user-specified conditions offers a promising
avenue for alleviating the production burden in 3D applications. Previous
studies required significant effort to realize the desired scene, owing to
limited control conditions. We propose a method for controlling and generating
3D scenes under multimodal conditions using partial images, layout information
represented in the top view, and text prompts. Combining these conditions to
generate a 3D scene involves the following significant difficulties: (1) the
creation of large datasets, (2) reflection on the interaction of multimodal
conditions, and (3) domain dependence of the layout conditions. We decompose
the process of 3D scene generation into 2D image generation from the given
conditions and 3D scene generation from 2D images. 2D image generation is
achieved by fine-tuning a pretrained text-to-image model with a small
artificial dataset of partial images and layouts, and 3D scene generation is
achieved by layout-conditioned depth estimation and neural radiance fields
(NeRF), thereby avoiding the creation of large datasets. The use of a common
representation of spatial information using 360-degree images allows for the
consideration of multimodal condition interactions and reduces the domain
dependence of the layout control. The experimental results qualitatively and
quantitatively demonstrated that the proposed method can generate 3D scenes in
diverse domains, from indoor to outdoor, according to multimodal conditions.

使用部分图像、顶视图的布局信息和文本提示来控制和生成多模态条件下的 3D 场景，通过预训练的文本到图像模型微调，以及布局条件的深度估计和神经辐射场（NeRF），实现从给定条件生成 2D 图像和从 2D 图像生成 3D 场景，避免创建庞大的数据集，得到了定性和定量的实验结果，证明了该方法能够根据多模态条件生成多领域的 3D 场景。