Monocular depth estimation from RGB images plays a pivotal role in 3D vision.
However, its accuracy can deteriorate in challenging environments such as
nighttime or adverse weather conditions. While long-wave infrared cameras offer
stable imaging in such challenging conditions, they are inherently
low-resolution, lacking rich texture and semantics as delivered by the RGB
image. Current methods focus solely on a single modality due to the
difficulties to identify and integrate faithful depth cues from both sources.
To address these issues, this paper presents a novel approach that identifies
and integrates dominant cross-modality depth features with a learning-based
framework. Concretely, we independently compute the coarse depth maps with
separate networks by fully utilizing the individual depth cues from each
modality. As the advantageous depth spreads across both modalities, we propose
a novel confidence loss steering a confidence predictor network to yield a
confidence map specifying latent potential depth areas. With the resulting
confidence map, we propose a multi-modal fusion network that fuses the final
depth in an end-to-end manner. Harnessing the proposed pipeline, our method
demonstrates the ability of robust depth estimation in a variety of difficult
scenarios. Experimental results on the challenging MS$^2$ and ViViD++ datasets
demonstrate the effectiveness and robustness of our method.

本文提出了一种新颖的方法，通过学习框架，识别和整合主导的跨模态深度特征，独立计算粗略深度图，并使用置信度损失和多模态融合网络进行端到端的深度估计，从而在各种困难情景中展示了鲁棒的深度估计能力。

揭示深度：一种多模态融合框架用于挑战场景

Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging  Scenarios

Depth estimation models have shown promising performance on clear scenes but
fail to generalize to adverse weather conditions due to illumination
variations, weather particles, etc. In this paper, we propose WeatherDepth, a
self-supervised robust depth estimation model with curriculum contrastive
learning, to tackle performance degradation in complex weather conditions.
Concretely, we first present a progressive curriculum learning scheme with
three simple-to-complex curricula to gradually adapt the model from clear to
relative adverse, and then to adverse weather scenes. It encourages the model
to gradually grasp beneficial depth cues against the weather effect, yielding
smoother and better domain adaption. Meanwhile, to prevent the model from
forgetting previous curricula, we integrate contrastive learning into different
curricula. Drawn the reference knowledge from the previous course, our strategy
establishes a depth consistency constraint between different courses towards
robust depth estimation in diverse weather. Besides, to reduce manual
intervention and better adapt to different models, we designed an adaptive
curriculum scheduler to automatically search for the best timing for course
switching. In the experiment, the proposed solution is proven to be easily
incorporated into various architectures and demonstrates state-of-the-art
(SoTA) performance on both synthetic and real weather datasets.

我们提出了 WeatherDepth，这是一个自监督的鲁棒深度估计模型，使用课程对比学习来应对复杂天气条件下性能下降的问题。通过逐步从晴朗到恶劣天气场景的简单到复杂的课程调整，鼓励模型逐渐掌握抵抗天气影响的有益深度线索，从而实现更平滑和更好的域适应。同时，我们在不同课程中引入对比学习，以防止模型遗忘先前的课程，并在不同课程之间建立深度一致性约束，从而实现对多样天气条件下鲁棒深度估计的目标。此外，为了减少人工干预并更好地适应不同模型，我们设计了一个自适应课程调度器，自动搜索最佳的课程切换时机。在实验中，我们的解决方案被证明可以轻松应用于各种架构，并在合成和真实天气数据集上展现了最先进的性能。