We propose a novel approach for 3D mesh reconstruction from multi-view
images. Our method takes inspiration from large reconstruction models like LRM
that use a transformer-based triplane generator and a Neural Radiance Field
(NeRF) model trained on multi-view images. However, in our method, we introduce
several important modifications that allow us to significantly enhance 3D
reconstruction quality. First of all, we examine the original LRM architecture
and find several shortcomings. Subsequently, we introduce respective
modifications to the LRM architecture, which lead to improved multi-view image
representation and more computationally efficient training. Second, in order to
improve geometry reconstruction and enable supervision at full image
resolution, we extract meshes from the NeRF field in a differentiable manner
and fine-tune the NeRF model through mesh rendering. These modifications allow
us to achieve state-of-the-art performance on both 2D and 3D evaluation
metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
Despite these superior results, our feed-forward model still struggles to
reconstruct complex textures, such as text and portraits on assets. To address
this, we introduce a lightweight per-instance texture refinement procedure.
This procedure fine-tunes the triplane representation and the NeRF color
estimation model on the mesh surface using the input multi-view images in just
4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
reconstruction of complex textures, such as text. Additionally, our approach
enables various downstream applications, including text- or image-to-3D
generation.

我们提出了一种新的方法，从多视角图像中进行 3D 网格重建。我们的方法借鉴了像 LRM 这样的大型重建模型，使用了基于 Transformer 的三面体生成器和在多视角图像上进行训练的神经辐射场（NeRF）模型。然而，在我们的方法中，我们引入了一些重要的修改，以显著提高 3D 重建质量。

GTR：通过几何和纹理细化改进大型三维重建模型

GTR: Improving Large 3D Reconstruction Models through Geometry and  Texture Refinement

We present InstantMesh, a feed-forward framework for instant 3D mesh
generation from a single image, featuring state-of-the-art generation quality
and significant training scalability. By synergizing the strengths of an
off-the-shelf multiview diffusion model and a sparse-view reconstruction model
based on the LRM architecture, InstantMesh is able to create diverse 3D assets
within 10 seconds. To enhance the training efficiency and exploit more
geometric supervisions, e.g, depths and normals, we integrate a differentiable
iso-surface extraction module into our framework and directly optimize on the
mesh representation. Experimental results on public datasets demonstrate that
InstantMesh significantly outperforms other latest image-to-3D baselines, both
qualitatively and quantitatively. We release all the code, weights, and demo of
InstantMesh, with the intention that it can make substantial contributions to
the community of 3D generative AI and empower both researchers and content
creators.

InstantMesh 是一种从单个图像进行即时 3D 网格生成的前向框架，具有最先进的生成质量和可扩展的训练能力。通过发挥现有多视角扩散模型和基于 LRM 架构的稀疏视图重建模型的优势，InstantMesh 能够在 10 秒内创建多样化的 3D 资源。通过将可微分的等值面提取模块集成到我们的框架中，并直接在网格表示上进行优化，以提高训练效率和利用更多几何监督，例如深度和法线。公共数据集上的实验结果表明，InstantMesh 在定性和定量上明显优于其他最新的图像到 3D 基准。我们发布了 InstantMesh 的所有代码，权重和演示，希望它能为 3D 生成 AI 社区做出巨大贡献，并赋予研究人员和内容创作者更多的力量。