Humans have the remarkable ability to construct consistent mental models of
an environment, even under limited or varying levels of illumination. We wish
to endow robots with this same capability. In this paper, we tackle the
challenge of constructing a photorealistic scene representation under poorly
illuminated conditions and with a moving light source. We approach the task of
modeling illumination as a learning problem, and utilize the developed
illumination model to aid in scene reconstruction. We introduce an innovative
framework that uses a data-driven approach, Neural Light Simulators (NeLiS), to
model and calibrate the camera-light system. Furthermore, we present DarkGS, a
method that applies NeLiS to create a relightable 3D Gaussian scene model
capable of real-time, photorealistic rendering from novel viewpoints. We show
the applicability and robustness of our proposed simulator and system in a
variety of real-world environments.

我们介绍了一种新的框架，使用数据驱动方法来模拟和校准摄像机 - 光源系统，并应用于创建一种可重新照明的 3D 高斯场景模型，从而实现了从新的视点进行实时、逼真渲染的能力，并展示了我们所提出的模拟器和系统在各种实际环境中的适用性和强健性。

DarkGS: 学习神经照明和 3D 高斯重照技术，用于黑暗环境下的机器人探索

DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for  Robotic Exploration in the Dark

The advancement of generative radiance fields has pushed the boundary of
3D-aware image synthesis. Motivated by the observation that a 3D object should
look realistic from multiple viewpoints, these methods introduce a multi-view
constraint as regularization to learn valid 3D radiance fields from 2D images.
Despite the progress, they often fall short of capturing accurate 3D shapes due
to the shape-color ambiguity, limiting their applicability in downstream tasks.
In this work, we address this ambiguity by proposing a novel shading-guided
generative implicit model that is able to learn a starkly improved shape
representation. Our key insight is that an accurate 3D shape should also yield
a realistic rendering under different lighting conditions. This multi-lighting
constraint is realized by modeling illumination explicitly and performing
shading with various lighting conditions. Gradients are derived by feeding the
synthesized images to a discriminator. To compensate for the additional
computational burden of calculating surface normals, we further devise an
efficient volume rendering strategy via surface tracking, reducing the training
and inference time by 24% and 48%, respectively. Our experiments on multiple
datasets show that the proposed approach achieves photorealistic 3D-aware image
synthesis while capturing accurate underlying 3D shapes. We demonstrate
improved performance of our approach on 3D shape reconstruction against
existing methods, and show its applicability on image relighting. Our code will
be released at this https URL

本研究提出了一种基于阴影引导的生成式隐式模型，该模型利用多角度光照约束对 3D 物体进行建模，使得在不同光照条件下产生逼真的渲染效果，从而改善先前方法在表达精确 3D 形状方面的局限性，并成功应用于图像重照和 3D 形状重建任务。

基于阴影引导的生成式隐式模型用于精确形状的 3D 感知图像合成

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware  Image Synthesis

We present a new generic method for shadow-aware multi-view satellite
photogrammetry of Earth Observation scenes. Our proposed method, the Shadow
Neural Radiance Field (S-NeRF) follows recent advances in implicit volumetric
representation learning. For each scene, we train S-NeRF using very high
spatial resolution optical images taken from known viewing angles. The learning
requires no labels or shape priors: it is self-supervised by an image
reconstruction loss. To accommodate for changing light source conditions both
from a directional light source (the Sun) and a diffuse light source (the sky),
we extend the NeRF approach in two ways. First, direct illumination from the
Sun is modeled via a local light source visibility field. Second, indirect
illumination from a diffuse light source is learned as a non-local color field
as a function of the position of the Sun. Quantitatively, the combination of
these factors reduces the altitude and color errors in shaded areas, compared
to NeRF. The S-NeRF methodology not only performs novel view synthesis and full
3D shape estimation, it also enables shadow detection, albedo synthesis, and
transient object filtering, without any explicit shape supervision.

我们提出了一种新的通用方法，用于阴影感知的多视角卫星摄影测量。我们的建议方法是采用隐式体积表示学习中的最新进展。通过训练 Shadow Neural Radiance Field (S-NeRF) 模型，我们能够实现对场景进行 3D 形状估算、阴影检测和透明物体过滤，而不需要任何显式形状监督。