Learning-based depth estimation has witnessed recent progress in multiple
directions; from self-supervision using monocular video to supervised methods
offering highest accuracy. Complementary to supervision, further boosts to
performance and robustness are gained by combining information from multiple
signals. In this paper we systematically investigate key trade-offs associated
with sensor and modality design choices as well as related model training
strategies. Our study leads us to a new method, capable of connecting
modality-specific advantages from polarisation, Time-of-Flight and
structured-light inputs. We propose a novel pipeline capable of estimating
depth from monocular polarisation for which we evaluate various training
signals. The inversion of differentiable analytic models thereby connects scene
geometry with polarisation and ToF signals and enables self-supervised and
cross-modal learning. In the absence of existing multimodal datasets, we
examine our approach with a custom-made multi-modal camera rig and collect
CroMo; the first dataset to consist of synchronized stereo polarisation,
indirect ToF and structured-light depth, captured at video rates. Extensive
experiments on challenging video scenes confirm both qualitative and
quantitative pipeline advantages where we are able to outperform competitive
monocular depth estimation method.

本文提出了一种使用多模式相机组合极化、ToF 和结构光信号的深度预测方法，该方法使用可微的分析模型将场景几何与极化和 ToF 等信号连接起来，实现自监督和跨模态学习，在自定义的 CroMo 数据集上获得了优异的性能表现。

CroMo: 单目深度估计的跨模态学习

CroMo: Cross-Modal Learning for Monocular Depth Estimation

It is well known that the passive stereo system cannot adapt well to weak
texture objects, e.g., white walls. However, these weak texture targets are
very common in indoor environments. In this paper, we present a novel stereo
system, which consists of two cameras (an RGB camera and an IR camera) and an
IR speckle projector. The RGB camera is used both for depth estimation and
texture acquisition. The IR camera and the speckle projector can form a
monocular structured-light (MSL) subsystem, while the two cameras can form a
binocular stereo subsystem. The depth map generated by the MSL subsystem can
provide external guidance for the stereo matching networks, which can improve
the matching accuracy significantly. In order to verify the effectiveness of
the proposed system, we build a prototype and collect a test dataset in indoor
scenes. The evaluation results show that the Bad 2.0 error of the proposed
system is 28.2% of the passive stereo system when the network RAFT is used. The
dataset and trained models are available at
this https URL

本文介绍了一种新型立体系统，利用 IR 相机和结构光方法提高了匹配精度，通过在室内场景中的测试数据证明了该系统的有效性。