Monocular depth estimation from RGB images plays a pivotal role in 3D vision.
However, its accuracy can deteriorate in challenging environments such as
nighttime or adverse weather conditions. While long-wave infrared cameras offer
stable imaging in such challenging conditions, they are inherently
low-resolution, lacking rich texture and semantics as delivered by the RGB
image. Current methods focus solely on a single modality due to the
difficulties to identify and integrate faithful depth cues from both sources.
To address these issues, this paper presents a novel approach that identifies
and integrates dominant cross-modality depth features with a learning-based
framework. Concretely, we independently compute the coarse depth maps with
separate networks by fully utilizing the individual depth cues from each
modality. As the advantageous depth spreads across both modalities, we propose
a novel confidence loss steering a confidence predictor network to yield a
confidence map specifying latent potential depth areas. With the resulting
confidence map, we propose a multi-modal fusion network that fuses the final
depth in an end-to-end manner. Harnessing the proposed pipeline, our method
demonstrates the ability of robust depth estimation in a variety of difficult
scenarios. Experimental results on the challenging MS$^2$ and ViViD++ datasets
demonstrate the effectiveness and robustness of our method.

本文提出了一种新颖的方法，通过学习框架，识别和整合主导的跨模态深度特征，独立计算粗略深度图，并使用置信度损失和多模态融合网络进行端到端的深度估计，从而在各种困难情景中展示了鲁棒的深度估计能力。

揭示深度：一种多模态融合框架用于挑战场景

Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging  Scenarios

This paper presents an end-to-end 3D convolutional network named
attention-based multi-modal fusion network (AMFNet) for the semantic scene
completion (SSC) task of inferring the occupancy and semantic labels of a
volumetric 3D scene from single-view RGB-D images. Compared with previous
methods which use only the semantic features extracted from RGB-D images, the
proposed AMFNet learns to perform effective 3D scene completion and semantic
segmentation simultaneously via leveraging the experience of inferring 2D
semantic segmentation from RGB-D images as well as the reliable depth cues in
spatial dimension. It is achieved by employing a multi-modal fusion
architecture boosted from 2D semantic segmentation and a 3D semantic completion
network empowered by residual attention blocks. We validate our method on both
the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results
show that our method respectively achieves the gains of 2.5% and 2.6% on the
synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the
state-of-the-art method.

本文提出了一种名为 AMFNet 的端到端三维卷积网络，用于从单视图 RGB-D 图像推断出体积三维场景的占用和语义标签。该网络通过融合 2D 语义分割和 3D 语义完成网络的经验以及空间维度的可靠深度线索，学习同时执行有效的 3D 场景完成和语义分割，验证表明在合成 SUNCG-RGBD 数据集和真实 NYUv2 数据集上都取得了明显的优越性。

基于注意力机制的多模态融合网络用于场景语义补全

Attention-based Multi-modal Fusion Network for Semantic Scene Completion

Semantic segmentation has made encouraging progress due to the success of
deep convolutional networks in recent years. Meanwhile, depth sensors become
prevalent nowadays, so depth maps can be acquired more easily. However, there
are few studies that focus on the RGB-D semantic segmentation task. Exploiting
the depth information effectiveness to improve performance is a challenge. In
this paper, we propose a novel solution named LDFNet, which incorporates
Luminance, Depth and Color information by a fusion-based network. It includes a
sub-network to process depth maps and employs luminance images to assist the
depth information in processes. LDFNet outperforms the other state-of-art
systems on the Cityscapes dataset, and its inference speed is faster than most
of the existing networks. The experimental results show the effectiveness of
the proposed multi-modal fusion network and its potential for practical
applications.

本文提出了一种名为 LDFNet 的新型解决方案，通过融合网络来综合利用亮度，深度和颜色信息，以提高 RGB-D 语义分割任务的性能，实验表明其潜力和适用性