This paper aims to recover object materials from posed images captured under
an unknown static lighting condition. Recent methods solve this task by
optimizing material parameters through differentiable physically based
rendering. However, due to the coupling between object geometry, materials, and
environment lighting, there is inherent ambiguity during the inverse rendering
process, preventing previous methods from obtaining accurate results. To
overcome this ill-posed problem, our key idea is to learn the material prior
with a generative model for regularizing the optimization process. We observe
that the general rendering equation can be split into diffuse and specular
shading terms, and thus formulate the material prior as diffusion models of
albedo and specular. Thanks to this design, our model can be trained using the
existing abundant 3D object data, and naturally acts as a versatile tool to
resolve the ambiguity when recovering material representations from RGB images.
In addition, we develop a coarse-to-fine training strategy that leverages
estimated materials to guide diffusion models to satisfy multi-view consistent
constraints, leading to more stable and accurate results. Extensive experiments
on real-world and synthetic datasets demonstrate that our approach achieves
state-of-the-art performance on material recovery. The code will be available
at this https URL

该研究旨在通过生成模型和扩散模型来恢复从 RGB 图像中的物体材料，并通过粗糙到精细的训练策略来提高稳定性和准确性。

IntrinsicAnything: 学习未知照明下反向渲染的扩散先验

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under  Unknown Illumination

Based on powerful text-to-image diffusion models, text-to-3D generation has
made significant progress in generating compelling geometry and appearance.
However, existing methods still struggle to recover high-fidelity object
materials, either only considering Lambertian reflectance, or failing to
disentangle BRDF materials from the environment lights. In this work, we
propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR
(\textbf{MATLABER}) that leverages a novel latent BRDF auto-encoder for
material generation. We train this auto-encoder with large-scale real-world
BRDF collections and ensure the smoothness of its latent space, which
implicitly acts as a natural distribution of materials. During appearance
modeling in text-to-3D generation, the latent BRDF embeddings, rather than BRDF
parameters, are predicted via a material network. Through exhaustive
experiments, our approach demonstrates the superiority over existing ones in
generating realistic and coherent object materials. Moreover, high-quality
materials naturally enable multiple downstream tasks such as relighting and
material editing. Code and model will be publicly available at
https://sheldontsui.github.io/projects/Matlaber.

基于强大的文本到图像扩散模型，文本到三维生成在生成引人注目的几何和外观方面取得了显著进展。本文提出了一种通过潜在 BRDF 自编码器生成材料的材料感知文本到三维生成方法（MATLABER）。通过在大规模真实世界的 BRDF 集合上训练这个自编码器，确保其潜在空间的平滑性，该空间在物料方面具有自然分布特性。在文本到三维生成中的外观建模过程中，通过材料网络预测潜在 BRDF 嵌入，而不是 BRDF 参数。通过详尽的实验，我们的方法在生成逼真连贯的物体材料方面表现出优越性。此外，高质量的材料自然地实现了多个下游任务，如重新照明和材料编辑。代码和模型将公开发布于 https://sheldontsui.github.io/projects/Matlaber。

MATLABER: 材料感知性文本到三维的隐含 BRDF 自动编码器

MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

Three-dimensional reconstruction of objects from shading information is a
challenging task in computer vision. As most of the approaches facing the
Photometric Stereo problem use simplified far-field assumptions, real-world
scenarios have essentially more complex physical effects that need to be
handled for accurately reconstructing the 3D shape. An increasing number of
methods have been proposed to address the problem when point light sources are
assumed to be nearby the target object. The proximity of the light sources
complicates the modeling of the image formation as the light behaviour requires
non-linear parameterisation to describe its propagation and attenuation.
To understand the capability of the approaches dealing with this near-field
scenario, the literature till now has used synthetically rendered photometric
images or minimal and very customised real-world data. In order to fill the gap
in evaluating near-field photometric stereo methods, we introduce LUCES the
first real-world 'dataset for near-fieLd point light soUrCe photomEtric Stereo'
of 14 objects of a varying of materials. A device counting 52 LEDs has been
designed to lit each object positioned 10 to 30 centimeters away from the
camera. Together with the raw images, in order to evaluate the 3D
reconstructions, the dataset includes both normal and depth maps for comparing
different features of the retrieved 3D geometry. Furthermore, we evaluate the
performance of the latest near-field Photometric Stereo algorithms on the
proposed dataset to assess the SOTA method with respect to actual close range
effects and object materials.

提出了第一个 LUCES 数据集，包括 52 个 LED 灯，对 14 个物体进行了近距离照明，并用于评估最新的近场物体光度立体算法的性能，以评估最新的近场光度立体算法在实际物体材料的近距离效果方面的表现。