Neural Radiance Fields (NeRFs) quickly evolved as the new de-facto standard
for the task of novel view synthesis when trained on a set of RGB images. In
this paper, we conduct a comprehensive evaluation of neural scene
representations, such as NeRFs, in the context of multi-modal learning.
Specifically, we present four different strategies of how to incorporate a
second modality, other than RGB, into NeRFs: (1) training from scratch
independently on both modalities; (2) pre-training on RGB and fine-tuning on
the second modality; (3) adding a second branch; and (4) adding a separate
component to predict (color) values of the additional modality. We chose
thermal imaging as second modality since it strongly differs from RGB in terms
of radiosity, making it challenging to integrate into neural scene
representations. For the evaluation of the proposed strategies, we captured a
new publicly available multi-view dataset, ThermalMix, consisting of six common
objects and about 360 RGB and thermal images in total. We employ cross-modality
calibration prior to data capturing, leading to high-quality alignments between
RGB and thermal images. Our findings reveal that adding a second branch to NeRF
performs best for novel view synthesis on thermal images while also yielding
compelling results on RGB. Finally, we also show that our analysis generalizes
to other modalities, including near-infrared images and depth maps. Project
page: this https URL

该论文通过综合评估神经场表示（如 NeRFs）在多模态学习中的作用，提出了四种不同的策略来将热成像作为第二模态融入 NeRFs，并发现添加第二分支对于热成像的新视角合成效果最佳，同时在 RGB 方面也产生了令人信服的结果，该分析结果还推广到包括近红外图像和深度图在内的其他模态。