Realistic conditional 3D scene synthesis significantly enhances and
accelerates the creation of virtual environments, which can also provide
extensive training data for computer vision and robotics research among other
applications. Diffusion models have shown great performance in related
applications, e.g., making precise arrangements of unordered sets. However,
these models have not been fully explored in floor-conditioned scene synthesis
problems. We present MiDiffusion, a novel mixed discrete-continuous diffusion
model architecture, designed to synthesize plausible 3D indoor scenes from
given room types, floor plans, and potentially pre-existing objects. We
represent a scene layout by a 2D floor plan and a set of objects, each defined
by its category, location, size, and orientation. Our approach uniquely
implements structured corruption across the mixed discrete semantic and
continuous geometric domains, resulting in a better conditioned problem for the
reverse denoising step. We evaluate our approach on the 3D-FRONT dataset. Our
experimental results demonstrate that MiDiffusion substantially outperforms
state-of-the-art autoregressive and diffusion models in floor-conditioned 3D
scene synthesis. In addition, our models can handle partial object constraints
via a corruption-and-masking strategy without task specific training. We show
MiDiffusion maintains clear advantages over existing approaches in scene
completion and furniture arrangement experiments.

MiDiffusion 是一种新的混合离散 - 连续扩散模型架构，用于根据房间类型、平面图和可能的预存在对象合成逼真的室内 3D 场景，相比于自回归模型和扩散模型，在楼层条件下的 3D 场景合成表现出明显的优势，并可通过破坏和遮蔽策略处理部分对象约束，无需专门训练。

混合扩散用于 3D 室内场景合成

Mixed Diffusion for 3D Indoor Scene Synthesis

Generating realistic 3D scenes is challenging due to the complexity of room
layouts and object geometries.We propose a sketch based knowledge enhanced
diffusion architecture (SEK) for generating customized, diverse, and plausible
3D scenes. SEK conditions the denoising process with a hand-drawn sketch of the
target scene and cues from an object relationship knowledge base. We first
construct an external knowledge base containing object relationships and then
leverage knowledge enhanced graph reasoning to assist our model in
understanding hand-drawn sketches. A scene is represented as a combination of
3D objects and their relationships, and then incrementally diffused to reach a
Gaussian distribution.We propose a 3D denoising scene transformer that learns
to reverse the diffusion process, conditioned by a hand-drawn sketch along with
knowledge cues, to regressively generate the scene including the 3D object
instances as well as their layout. Experiments on the 3D-FRONT dataset show
that our model improves FID, CKL by 17.41%, 37.18% in 3D scene generation and
FID, KID by 19.12%, 20.06% in 3D scene completion compared to the nearest
competitor DiffuScene.

基于手绘草图和对象关系知识的增强扩散架构 (SEK) 用于生成个性化、多样化和可信的 3D 场景，通过外部知识库和图推理辅助模型理解草图，利用 3D 去噪场景转换器生成包括对象实例和布局的场景。与竞争者相比，我们的模型在 3D 场景生成和完成中提高了 FID、CKL 17.41%、37.18% 以及 FID、KID 19.12%、20.06%。