Depth prediction is a critical problem in robotics applications especially
autonomous driving. Generally, depth prediction based on binocular stereo
matching and fusion of monocular image and laser point cloud are two mainstream
methods. However, the former usually suffers from overfitting while building
cost volume, and the latter has a limited generalization due to the lack of
geometric constraint. To solve these problems, we propose a novel multimodal
neural network, namely UAMD-Net, for dense depth completion based on fusion of
binocular stereo matching and the weak constrain from the sparse point clouds.
Specifically, the sparse point clouds are converted to sparse depth map and
sent to the multimodal feature encoder (MFE) with binocular image, constructing
a cross-modal cost volume. Then, it will be further processed by the multimodal
feature aggregator (MFA) and the depth regression layer. Furthermore, the
existing multimodal methods ignore the problem of modal dependence, that is,
the network will not work when a certain modal input has a problem. Therefore,
we propose a new training strategy called Modal-dropout which enables the
network to be adaptively trained with multiple modal inputs and inference with
specific modal inputs. Benefiting from the flexible network structure and
adaptive training method, our proposed network can realize unified training
under various modal input conditions. Comprehensive experiments conducted on
KITTI depth completion benchmark demonstrate that our method produces robust
results and outperforms other state-of-the-art methods.

提出了一种新的多模态神经网络 UAMD-Net，通过融合双目立体匹配和稀疏点云的弱约束进行深度完善，并使用新的训练策略 Modal-dropout 使得网络能够适应多种模态输入条件，该方法在 KITTI 深度完善基准测试中的实验表明其能够产生鲁棒的结果，并优于其他最先进的方法。