In this paper, we formulate a potentially valuable panoramic depth completion
(PDC) task as panoramic 3D cameras often produce 360{\deg} depth with missing
data in complex scenes. Its goal is to recover dense panoramic depths from raw
sparse ones and panoramic RGB images. To deal with the PDC task, we train a
deep network that takes both depth and image as inputs for the dense panoramic
depth recovery. However, it needs to face a challenging optimization problem of
the network parameters due to its non-convex objective function. To address
this problem, we propose a simple yet effective approach termed M{^3}PT:
multi-modal masked pre-training. Specifically, during pre-training, we
simultaneously cover up patches of the panoramic RGB image and sparse depth by
shared random mask, then reconstruct the sparse depth in the masked regions. To
our best knowledge, it is the first time that we show the effectiveness of
masked pre-training in a multi-modal vision task, instead of the single-modal
task resolved by masked autoencoders (MAE). Different from MAE where
fine-tuning completely discards the decoder part of pre-training, there is no
architectural difference between the pre-training and fine-tuning stages in our
M$^{3}$PT as they only differ in the prediction density, which potentially
makes the transfer learning more convenient and effective. Extensive
experiments verify the effectiveness of M{^3}PT on three panoramic datasets.
Notably, we improve the state-of-the-art baselines by averagely 26.2% in RMSE,
51.7% in MRE, 49.7% in MAE, and 37.5% in RMSElog on three benchmark datasets.

本文提出了一种称为 M {^3} PT 的简单而有效的方法，用于 360 度深度图像补全，使用 multi-modal masked pre-training 实现，从而在三个 360 度数据集上取得了显著的改进。