This paper aims to recover object materials from posed images captured under
an unknown static lighting condition. Recent methods solve this task by
optimizing material parameters through differentiable physically based
rendering. However, due to the coupling between object geometry, materials, and
environment lighting, there is inherent ambiguity during the inverse rendering
process, preventing previous methods from obtaining accurate results. To
overcome this ill-posed problem, our key idea is to learn the material prior
with a generative model for regularizing the optimization process. We observe
that the general rendering equation can be split into diffuse and specular
shading terms, and thus formulate the material prior as diffusion models of
albedo and specular. Thanks to this design, our model can be trained using the
existing abundant 3D object data, and naturally acts as a versatile tool to
resolve the ambiguity when recovering material representations from RGB images.
In addition, we develop a coarse-to-fine training strategy that leverages
estimated materials to guide diffusion models to satisfy multi-view consistent
constraints, leading to more stable and accurate results. Extensive experiments
on real-world and synthetic datasets demonstrate that our approach achieves
state-of-the-art performance on material recovery. The code will be available
at this https URL

该研究旨在通过生成模型和扩散模型来恢复从 RGB 图像中的物体材料，并通过粗糙到精细的训练策略来提高稳定性和准确性。

IntrinsicAnything: 学习未知照明下反向渲染的扩散先验

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under  Unknown Illumination

We introduce Stanford-ORB, a new real-world 3D Object inverse Rendering
Benchmark. Recent advances in inverse rendering have enabled a wide range of
real-world applications in 3D content generation, moving rapidly from research
and commercial use cases to consumer devices. While the results continue to
improve, there is no real-world benchmark that can quantitatively assess and
compare the performance of various inverse rendering methods. Existing
real-world datasets typically only consist of the shape and multi-view images
of objects, which are not sufficient for evaluating the quality of material
recovery and object relighting. Methods capable of recovering material and
lighting often resort to synthetic data for quantitative evaluation, which on
the other hand does not guarantee generalization to complex real-world
environments. We introduce a new dataset of real-world objects captured under a
variety of natural scenes with ground-truth 3D scans, multi-view images, and
environment lighting. Using this dataset, we establish the first comprehensive
real-world evaluation benchmark for object inverse rendering tasks from
in-the-wild scenes, and compare the performance of various existing methods.
All data, code, and models can be accessed at this https URL

我们引入了 Stanford-ORB，一个新的真实世界的 3D 物体逆向渲染基准。我们通过提供一个新的真实世界的数据集，并使用该数据集建立了首个全面的真实世界评估基准，评估对象逆向渲染任务在野外场景的性能，并比较了各种现有方法的表现。