Traditional image-to-3D models often struggle with scenes containing multiple
objects due to biases and occlusion complexities. To address this challenge, we
present REPARO, a novel approach for compositional 3D asset generation from
single images. REPARO employs a two-step process: first, it extracts individual
objects from the scene and reconstructs their 3D meshes using off-the-shelf
image-to-3D models; then, it optimizes the layout of these meshes through
differentiable rendering techniques, ensuring coherent scene composition. By
integrating optimal transport-based long-range appearance loss term and
high-level semantic loss term in the differentiable rendering, REPARO can
effectively recover the layout of 3D assets. The proposed method can
significantly enhance object independence, detail accuracy, and overall scene
coherence. Extensive evaluation of multi-object scenes demonstrates that our
REPARO offers a comprehensive approach to address the complexities of
multi-object 3D scene generation from single images.

REPARO 是一种从单个图像生成多个物体的三维模型的新方法，通过提取物体、重构三维网格、优化布局以及使用可微渲染技术，它能够有效地恢复三维资产的布局，达到高度的独立性、精确性和整体场景的一致性，从而解决了从单个图像生成多个物体的三维场景的复杂性问题。

REPARO: 采用可微分 3D 布局对齐生成复合式 3D 资源

REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout  Alignment

Obtaining 3D object representations is important for creating photo-realistic
simulations and for collecting AR and VR assets. Neural fields have shown their
effectiveness in learning a continuous volumetric representation of a scene
from 2D images, but acquiring object representations from these models with
weak supervision remains an open challenge. In this paper we introduce LaTeRF,
a method for extracting an object of interest from a scene given 2D images of
the entire scene, known camera poses, a natural language description of the
object, and a set of point-labels of object and non-object points in the input
images. To faithfully extract the object from the scene, LaTeRF extends the
NeRF formulation with an additional `objectness' probability at each 3D point.
Additionally, we leverage the rich latent space of a pre-trained CLIP model
combined with our differentiable object renderer, to inpaint the occluded parts
of the object. We demonstrate high-fidelity object extraction on both synthetic
and real-world datasets and justify our design choices through an extensive
ablation study.

本研究提出了 LaTeRF 方法，通过引入 “对象性” 概率，扩展 NeRF 公式，结合自然语言描述、点标签等信息从场景中提取出感兴趣的物体，并结合预训练的 CLIP 模型和可微分对象渲染器来修复物体的遮挡部分。实验结果表明该方法在合成和真实数据集上均能实现高保真物体提取。