We introduce HouseCrafter, a novel approach that can lift a floorplan into a
complete large 3D indoor scene (e.g., a house). Our key insight is to adapt a
2D diffusion model, which is trained on web-scale images, to generate
consistent multi-view color (RGB) and depth (D) images across different
locations of the scene. Specifically, the RGB-D images are generated
autoregressively in a batch-wise manner along sampled locations based on the
floorplan, where previously generated images are used as condition to the
diffusion model to produce images at nearby locations. The global floorplan and
attention design in the diffusion model ensures the consistency of the
generated images, from which a 3D scene can be reconstructed. Through extensive
evaluation on the 3D-Front dataset, we demonstrate that HouseCraft can generate
high-quality house-scale 3D scenes. Ablation studies also validate the
effectiveness of different design choices. We will release our code and model
weights. Project page: this https URL

HouseCrafter 使用 2D 扩散模型生成与全景平面图一致的多视角 RGB-D 图像，从而重建高质量的 3D 室内场景。

HouseCrafter：使用 2D 扩散模型将平面图提升为 3D 场景

HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model

We present a new framework to reconstruct holistic 3D indoor scenes including
both room background and indoor objects from single-view images. Existing
methods can only produce 3D shapes of indoor objects with limited geometry
quality because of the heavy occlusion of indoor scenes. To solve this, we
propose an instance-aligned implicit function (InstPIFu) for detailed object
reconstruction. Combining with instance-aligned attention module, our method is
empowered to decouple mixed local features toward the occluded instances.
Additionally, unlike previous methods that simply represents the room
background as a 3D bounding box, depth map or a set of planes, we recover the
fine geometry of the background via implicit representation. Extensive
experiments on the SUN RGB-D, Pix3D, 3D-FUTURE, and 3D-FRONT datasets
demonstrate that our method outperforms existing approaches in both background
and foreground object reconstruction. Our code and model will be made publicly
available.

我们提出了一种新的框架来重建立体的室内场景，包括从单视图图像中提取的背景和室内物体。我们的方法是将实例对齐的隐式函数（InstPIFu）与对齐注意力模块相结合，从而使我们的方法能够解耦合并的局部特征以实现物体的详细重构和恢复背景的细致几何结构，实验证明了我们方法在前景和背景物体重构方面的优越性。

室内场景高保真单视角整体重建

Towards High-Fidelity Single-view Holistic Reconstruction of Indoor  Scenes

In this paper we propose a neural message passing approach to augment an
input 3D indoor scene with new objects matching their surroundings. Given an
input, potentially incomplete, 3D scene and a query location, our method
predicts a probability distribution over object types that fit well in that
location. Our distribution is predicted though passing learned messages in a
dense graph whose nodes represent objects in the input scene and edges
represent spatial and structural relationships. By weighting messages through
an attention mechanism, our method learns to focus on the most relevant
surrounding scene context to predict new scene objects. We found that our
method significantly outperforms state-of-the-art approaches in terms of
correctly predicting objects missing in a scene based on our experiments in the
SUNCG dataset. We also demonstrate other applications of our method, including
context-based 3D object recognition and iterative scene generation.

本文提出了一种神经传递信息的方法来与其环境相匹配的新物体进行增强，该方法能够预测适合位置的物体类型的概率分布，并在密集图中传递学习消息来处理物体相之间的空间和结构关系，通过注意机制加权消息，在 SUNCG 数据集中比其他方法更准确地预测场景中缺失的物体，并展示了基于此方法的其他应用，包括基于上下文的 3D 对象识别和迭代场景生成。