Solving image-to-3D from a single view is an ill-posed problem, and current
neural reconstruction methods addressing it through diffusion models still rely
on scene-specific optimization, constraining their generalization capability.
To overcome the limitations of existing approaches regarding generalization and
consistency, we introduce a novel neural rendering technique. Our approach
employs the signed distance function as the surface representation and
incorporates generalizable priors through geometry-encoding volumes and
HyperNetworks. Specifically, our method builds neural encoding volumes from
generated multi-view inputs. We adjust the weights of the SDF network
conditioned on an input image at test-time to allow model adaptation to novel
scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts
derived from the synthesized views, we propose the use of a volume transformer
module to improve the aggregation of image features instead of processing each
viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we
avoid the bottleneck of scene-specific optimization and maintain consistency
across the images generated from multiple viewpoints. Our experiments show the
advantages of our proposed approach with consistent results and rapid
generation.

从单一视角解决图像到三维的问题是一个不适定问题，现有的神经重建方法依靠场景特定的优化限制其泛化能力。为了克服现有方法在泛化和一致性方面的局限性，我们引入了一种新颖的神经渲染技术。我们的方法通过几何编码体积和超网络，采用有符号距离函数作为表面表示，并从生成的多视图输入构建神经编码体积，在测试时根据输入图像调整 SDF 网络的权重，通过超网络以前馈方式实现对新场景的模型适应。为了减轻合成视图产生的伪影，我们提出了一个体素变换模块来改善图像特征的聚合，而不是单独处理每个视点。通过我们提出的 Hyper-VolTran 方法，我们避免了场景特定优化的瓶颈，并保持从多个视点生成的图像的一致性。我们的实验证明了我们提出的方法的优越性，具有一致的结果和快速生成。

Hyper-VolTran：基于超网络的快速且通用的一次性图像到 3D 物体结构转换

Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object  Structure via HyperNetworks

Object recognition has seen significant progress in the image domain, with
focus primarily on 2D perception. We propose to leverage existing large-scale
datasets of 3D models to understand the underlying 3D structure of objects seen
in an image by constructing a CAD-based representation of the objects and their
poses. We present Mask2CAD, which jointly detects objects in real-world images
and for each detected object, optimizes for the most similar CAD model and its
pose. We construct a joint embedding space between the detected regions of an
image corresponding to an object and 3D CAD models, enabling retrieval of CAD
models for an input RGB image. This produces a clean, lightweight
representation of the objects in an image; this CAD-based representation
ensures a valid, efficient shape representation for applications such as
content creation or interactive scenarios, and makes a step towards
understanding the transformation of real-world imagery to a synthetic domain.
Experiments on real-world images from Pix3D demonstrate the advantage of our
approach in comparison to state of the art. To facilitate future research, we
additionally propose a new image-to-3D baseline on ScanNet which features
larger shape diversity, real-world occlusions, and challenging image views.

本文提出了一种基于 CAD 模型的方法，结合现有的大规模三维模型数据集，构建了一种可以检测现实世界图像中物体的 CAD 模型表示方法，并且证明了与现有技术相比，该方法在真实世界图像中有着优势。