Reconstructing textured meshes from colored point clouds is an important but
challenging task in 3D graphics and vision. Most existing methods predict
colors as implicit functions in 3D or UV space, suffering from blurry textures
or the lack of generalization capability. Addressing this, we propose
PointDreamer, a novel framework for textured mesh reconstruction from colored
point cloud. It produces meshes with enhanced fidelity and clarity by 2D image
inpainting, taking advantage of the mature techniques and massive data of 2D
vision. Specifically, we first project the input point cloud into 2D space to
generate sparse multi-view images, and then inpaint empty pixels utilizing a
pre-trained 2D diffusion model. Next, we design a novel Non-Border-First
strategy to unproject the colors of the inpainted dense images back to 3D
space, thus obtaining the final textured mesh. In this way, our PointDreamer
works in a zero-shot manner, requiring no extra training. Extensive qualitative
and quantitative experiments on various synthetic and real-scanned datasets
show the SoTA performance of PointDreamer, by significantly outperforming
baseline methods with 30\% improvement in LPIPS score (from 0.118 to 0.068).
Code at: this https URL

从有颜色的点云中重建纹理网格是三维图形和视觉中一个重要且具有挑战性的任务。本文提出了一种新型的框架 PointDreamer，通过二维图像修复来提高纹理网格的保真度和清晰度，实现了从有颜色的点云中重建纹理网格的目标。

PointDreamer: 从彩色点云中零射线三维纹理网格重建通过二维修复

PointDreamer: Zero-shot 3D Textured Mesh Reconstruction from Colored  Point Cloud by 2D Inpainting

2D diffusion model, which often contains unwanted baked-in shading effects
and results in unrealistic rendering effects in the downstream applications.
Generating Physically Based Rendering (PBR) materials instead of just RGB
textures would be a promising solution. However, directly distilling the PBR
material parameters from 2D diffusion models still suffers from incorrect
material decomposition, such as baked-in shading effects in albedo. We
introduce DreamMat, an innovative approach to resolve the aforementioned
problem, to generate high-quality PBR materials from text descriptions. We find
out that the main reason for the incorrect material distillation is that
large-scale 2D diffusion models are only trained to generate final shading
colors, resulting in insufficient constraints on material decomposition during
distillation. To tackle this problem, we first finetune a new light-aware 2D
diffusion model to condition on a given lighting environment and generate the
shading results on this specific lighting condition. Then, by applying the same
environment lights in the material distillation, DreamMat can generate
high-quality PBR materials that are not only consistent with the given geometry
but also free from any baked-in shading effects in albedo. Extensive
experiments demonstrate that the materials produced through our methods exhibit
greater visual appeal to users and achieve significantly superior rendering
quality compared to baseline methods, which are preferable for downstream tasks
such as game and film production.

通过 DreamMat 方法从文本描述生成高质量 PBR 材料，解决了从 2D 扩散模型中提取 PBR 材料参数时的错误材料解析问题，并取得了显著的渲染质量优势。