Existing color-guided depth super-resolution (DSR) approaches require paired
RGB-D data as training samples where the RGB image is used as structural
guidance to recover the degraded depth map due to their geometrical similarity.
However, the paired data may be limited or expensive to be collected in actual
testing environment. Therefore, we explore for the first time to learn the
cross-modality knowledge at training stage, where both RGB and depth modalities
are available, but test on the target dataset, where only single depth modality
exists. Our key idea is to distill the knowledge of scene structural guidance
from RGB modality to the single DSR task without changing its network
architecture. Specifically, we construct an auxiliary depth estimation (DE)
task that takes an RGB image as input to estimate a depth map, and train both
DSR task and DE task collaboratively to boost the performance of DSR. Upon
this, a cross-task interaction module is proposed to realize bilateral cross
task knowledge transfer. First, we design a cross-task distillation scheme that
encourages DSR and DE networks to learn from each other in a teacher-student
role-exchanging fashion. Then, we advance a structure prediction (SP) task that
provides extra structure regularization to help both DSR and DE networks learn
more informative structure representations for depth recovery. Extensive
experiments demonstrate that our scheme achieves superior performance in
comparison with other DSR methods.

在没有 RGB-D 成对数据的情况下，为第一次探索如何在训练阶段学习跨模态知识，并在单一深度模态的目标数据集上进行测试。通过构建一个辅助深度估计任务，实现了 RGB 模态中的场景结构指导知识到单一深度模态下的 DSR 任务的跨任务交互。广泛实验表明，该方案相较于其他 DSR 方法表现优异。