Different outdoor illumination conditions drastically alter the appearance of
urban scenes, and they can harm the performance of image-based robot perception
systems if not seen during training. Camera simulation provides a
cost-effective solution to create a large dataset of images captured under
different lighting conditions. Towards this goal, we propose LightSim, a neural
lighting camera simulation system that enables diverse, realistic, and
controllable data generation. LightSim automatically builds lighting-aware
digital twins at scale from collected raw sensor data and decomposes the scene
into dynamic actors and static background with accurate geometry, appearance,
and estimated scene lighting. These digital twins enable actor insertion,
modification, removal, and rendering from a new viewpoint, all in a
lighting-aware manner. LightSim then combines physically-based and learnable
deferred rendering to perform realistic relighting of modified scenes, such as
altering the sun location and modifying the shadows or changing the sun
brightness, producing spatially- and temporally-consistent camera videos. Our
experiments show that LightSim generates more realistic relighting results than
prior work. Importantly, training perception models on data generated by
LightSim can significantly improve their performance.

研究提出了一个名为 LightSim 的神经照明相机模拟系统，用于在不同照明条件下产生多样化、逼真且可控的数据生成，实验结果显示 LightSim 产生的实际光照效果比以往工作更逼真，并且基于 LightSim 生成的数据可以显著提高感知模型的性能。