The default strategy for training single-view Large Reconstruction Models
(LRMs) follows the fully supervised route using large-scale datasets of
synthetic 3D assets or multi-view captures. Although these resources simplify
the training procedure, they are hard to scale up beyond the existing datasets
and they are not necessarily representative of the real distribution of object
shapes. To address these limitations, in this paper, we introduce Real3D, the
first LRM system that can be trained using single-view real-world images.
Real3D introduces a novel self-training framework that can benefit from both
the existing synthetic data and diverse single-view real images. We propose two
unsupervised losses that allow us to supervise LRMs at the pixel- and
semantic-level, even for training examples without ground-truth 3D or novel
views. To further improve performance and scale up the image data, we develop
an automatic data curation approach to collect high-quality examples from
in-the-wild images. Our experiments show that Real3D consistently outperforms
prior work in four diverse evaluation settings that include real and synthetic
data, as well as both in-domain and out-of-domain shapes. Code and model can be
found here: this https URL

该研究介绍了 Real3D，使用单视图真实世界图像训练的第一个大型重建模型系统，通过引入自主训练框架和无监督的损失函数，同时利用现有的合成数据和多样化的单视图真实图像，提高了性能和扩大了图像数据的规模，实验结果表明 Real3D 在不同的评估设置中优于以前的工作。

Real3D：用真实世界图像扩展大型重建模型

Real3D: Scaling Up Large Reconstruction Models with Real-World Images

Large Reconstruction Models have made significant strides in the realm of
automated 3D content generation from single or multiple input images. Despite
their success, these models often produce 3D meshes with geometric
inaccuracies, stemming from the inherent challenges of deducing 3D shapes
solely from image data. In this work, we introduce a novel framework, the Large
Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud
data to enhance the fidelity of generated 3D meshes. Our methodology begins
with the development of a point-cloud-based network that effectively generates
precise and meaningful latent tri-planes, laying the groundwork for accurate 3D
mesh reconstruction. Building upon this, our Image-Point-Cloud Feature
Alignment technique processes a single input image, aligning to the latent
tri-planes to imbue image features with robust 3D information. This process not
only enriches the image features but also facilitates the production of
high-fidelity 3D meshes without the need for multi-view input, significantly
reducing geometric distortions. Our approach achieves state-of-the-art
high-fidelity 3D mesh reconstruction from a single image in just 6 seconds, and
experiments on various datasets demonstrate its effectiveness.

大规模重建模型在自动化三维内容生成方面取得了重大突破。然而，这些模型常常产生具有几何不精确性的三维网格，这是由于仅从图像数据推断三维形状的固有挑战所引起的。在本文中，我们介绍了一种新颖的框架，即大型图像与点云对齐模型（LAM3D），它利用三维点云数据增强了生成的三维网格的保真度。我们的方法从开发基于点云的网络开始，有效地生成精确而有意义的潜在三平面，为准确的三维网格重建奠定了基础。在此基础上，我们的图像 - 点云特征对齐技术通过处理单个输入图像，将其与潜在三平面对齐，使图像特征具备强大的三维信息。这个过程不仅丰富了图像特征，还有助于产生高保真度的三维网格，而无需多视图输入，大大降低了几何变形。我们的方法在仅 6 秒内实现了单个图像的最先进高保真度三维网格重建，并在各种数据集上的实验证明了它的有效性。

LAM3D：用于单图像三维重建的大规模图像 - 点云对齐模型

LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction  from Single Image

We propose MeshLRM, a novel LRM-based approach that can reconstruct a
high-quality mesh from merely four input images in less than one second.
Different from previous large reconstruction models (LRMs) that focus on
NeRF-based reconstruction, MeshLRM incorporates differentiable mesh extraction
and rendering within the LRM framework. This allows for end-to-end mesh
reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering.
Moreover, we improve the LRM architecture by simplifying several complex
designs in previous LRMs. MeshLRM's NeRF initialization is sequentially trained
with low- and high-resolution images; this new LRM training strategy enables
significantly faster convergence and thereby leads to better quality with less
compute. Our approach achieves state-of-the-art mesh reconstruction from
sparse-view inputs and also allows for many downstream applications, including
text-to-3D and single-image-to-3D generation. Project page:
this https URL

提出了 MeshLRM，一种基于 LRM 的新方法，在不到一秒的时间内仅使用四个输入图像进行高质量网格的重建。与以往的大型重建模型（LRMs）不同，MeshLRM 在 LRM 框架中融入了可微分网格提取和渲染，可通过微调预训练的 NeRF LRM 进行端到端的网格重建。此外，通过简化先前 LRMs 中的多个复杂设计，我们改进了 LRM 架构。MeshLRM 的 NeRF 初始化使用低分辨率和高分辨率图像进行顺序训练；这种新的 LRM 训练策略实现了显著更快的收敛速度，从而在更少的计算量下获得更好的质量。我们的方法在稀疏视图输入下实现了最先进的网格重建，并且还可应用于很多下游应用，包括文本到三维和单图像到三维的生成。项目页面：此链接。