We propose a novel approach for 3D mesh reconstruction from multi-view
images. Our method takes inspiration from large reconstruction models like LRM
that use a transformer-based triplane generator and a Neural Radiance Field
(NeRF) model trained on multi-view images. However, in our method, we introduce
several important modifications that allow us to significantly enhance 3D
reconstruction quality. First of all, we examine the original LRM architecture
and find several shortcomings. Subsequently, we introduce respective
modifications to the LRM architecture, which lead to improved multi-view image
representation and more computationally efficient training. Second, in order to
improve geometry reconstruction and enable supervision at full image
resolution, we extract meshes from the NeRF field in a differentiable manner
and fine-tune the NeRF model through mesh rendering. These modifications allow
us to achieve state-of-the-art performance on both 2D and 3D evaluation
metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
Despite these superior results, our feed-forward model still struggles to
reconstruct complex textures, such as text and portraits on assets. To address
this, we introduce a lightweight per-instance texture refinement procedure.
This procedure fine-tunes the triplane representation and the NeRF color
estimation model on the mesh surface using the input multi-view images in just
4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
reconstruction of complex textures, such as text. Additionally, our approach
enables various downstream applications, including text- or image-to-3D
generation.

我们提出了一种新的方法，从多视角图像中进行 3D 网格重建。我们的方法借鉴了像 LRM 这样的大型重建模型，使用了基于 Transformer 的三面体生成器和在多视角图像上进行训练的神经辐射场（NeRF）模型。然而，在我们的方法中，我们引入了一些重要的修改，以显著提高 3D 重建质量。