We introduce GeoWizard, a new generative foundation model designed for
estimating geometric attributes, e.g., depth and normals, from single images.
While significant research has already been conducted in this area, the
progress has been substantially limited by the low diversity and poor quality
of publicly available datasets. As a result, the prior works either are
constrained to limited scenarios or suffer from the inability to capture
geometric details. In this paper, we demonstrate that generative models, as
opposed to traditional discriminative models (e.g., CNNs and Transformers), can
effectively address the inherently ill-posed problem. We further show that
leveraging diffusion priors can markedly improve generalization, detail
preservation, and efficiency in resource usage. Specifically, we extend the
original stable diffusion model to jointly predict depth and normal, allowing
mutual information exchange and high consistency between the two
representations. More importantly, we propose a simple yet effective strategy
to segregate the complex data distribution of various scenes into distinct
sub-distributions. This strategy enables our model to recognize different scene
layouts, capturing 3D geometry with remarkable fidelity. GeoWizard sets new
benchmarks for zero-shot depth and normal prediction, significantly enhancing
many downstream applications such as 3D reconstruction, 2D content creation,
and novel viewpoint synthesis.

我们介绍了一种新的生成式基础模型 GeoWizard，用于从单幅图像中估计几何属性，例如深度和法线。我们展示了生成模型在解决固有的不适定问题方面的有效性，通过扩展原始的稳定扩散模型同时预测深度和法线，我们可以捕捉具有显著保真度的 3D 几何信息。GeoWizard 在零样本深度和法线预测方面树立了新的基准，大大提升了许多下游应用程序的性能。

GeoWizard：从单张图像中释放 3D 几何估计的扩散先验

GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation  from a Single Image

Learning descriptive 3D features is crucial for understanding 3D scenes with
diverse objects and complex structures. However, it is usually unknown whether
important geometric attributes and scene context obtain enough emphasis in an
end-to-end trained 3D scene understanding network. To guide 3D feature learning
toward important geometric attributes and scene context, we explore the help of
textual scene descriptions. Given some free-form descriptions paired with 3D
scenes, we extract the knowledge regarding the object relationships and object
attributes. We then inject the knowledge to 3D feature learning through three
classification-based auxiliary tasks. This language-assisted training can be
combined with modern object detection and instance segmentation methods to
promote 3D semantic scene understanding, especially in a label-deficient
regime. Moreover, the 3D feature learned with language assistance is better
aligned with the language features, which can benefit various 3D-language
multimodal tasks. Experiments on several benchmarks of 3D-only and 3D-language
tasks demonstrate the effectiveness of our language-assisted 3D feature
learning. Code is available at
this https URL

通过文本场景描述信息辅助 3D 特征学习，进而提升三维语义场景理解的效果，并构建更好的语言与三维结构的多模态任务。

基于语言辅助的三维特征学习，用于语义场景理解

Language-Assisted 3D Feature Learning for Semantic Scene Understanding

Objects are made of parts, each with distinct geometry, physics,
functionality, and affordances. Developing such a distributed, physical,
interpretable representation of objects will facilitate intelligent agents to
better explore and interact with the world. In this paper, we study physical
primitive decomposition---understanding an object through its components, each
with physical and geometric attributes. As annotated data for object parts and
physics are rare, we propose a novel formulation that learns physical
primitives by explaining both an object's appearance and its behaviors in
physical events. Our model performs well on block towers and tools in both
synthetic and real scenarios; we also demonstrate that visual and physical
observations often provide complementary signals. We further present ablation
and behavioral studies to better understand our model and contrast it with
human performance.

本篇研究论文提出了一种新颖的物理基元分解方法，通过解释物体的外观和物理行为来学习其组成部分，以便于智能体更好地探索和交互世界。

物理基元分解

Physical Primitive Decomposition

Inference of topological and geometric attributes of a hidden manifold from
its point data is a fundamental problem arising in many scientific studies and
engineering applications. In this paper we present an algorithm to compute a
set of loops from a point data that presumably sample a smooth manifold
$M\subset \mathbb{R}^d$. These loops approximate a {\em shortest} basis of the
one dimensional homology group $H_1(M)$ over coefficients in finite field
$\mathbb{Z}_2$. Previous results addressed the issue of computing the rank of
the homology groups from point data, but there is no result on approximating
the shortest basis of a manifold from its point sample. In arriving our result,
we also present a polynomial time algorithm for computing a shortest basis of
$H_1(K)$ for any finite {\em simplicial complex} $K$ whose edges have
non-negative weights.

本文提出了一种算法，可从样本点推断出隐藏流形的拓扑和几何属性，并计算一个基的集合，近似于稳定流形中一维同调群的最短基。