The raw depth image captured by indoor depth sensors usually has an extensive
range of missing depth values due to inherent limitations such as the inability
to perceive transparent objects and the limited distance range. The incomplete
depth map with missing values burdens many downstream vision tasks, and a
rising number of depth completion methods have been proposed to alleviate this
issue. While most existing methods can generate accurate dense depth maps from
sparse and uniformly sampled depth maps, they are not suitable for
complementing large contiguous regions of missing depth values, which is common
and critical in images captured in indoor environments. To overcome these
challenges, we design a novel two-branch end-to-end fusion network named
RDFC-GAN, which takes a pair of RGB and incomplete depth images as input to
predict a dense and completed depth map. The first branch employs an
encoder-decoder structure, by adhering to the Manhattan world assumption and
utilizing normal maps from RGB-D information as guidance, to regress the local
dense depth values from the raw depth map. In the other branch, we propose an
RGB-depth fusion CycleGAN to transfer the RGB image to the fine-grained
textured depth map. We adopt adaptive fusion modules named W-AdaIN to propagate
the features across the two branches, and we append a confidence fusion head to
fuse the two outputs of the branches for the final depth map. Extensive
experiments on NYU-Depth V2 and SUN RGB-D demonstrate that our proposed method
clearly improves the depth completion performance, especially in a more
realistic setting of indoor environments, with the help of our proposed pseudo
depth maps in training.

使用 RDFC-GAN 模型基于 Manhattan world 假设将 RGB-D 信息转换为具有高纹理的深度图，并通过自适应融合模块 W-AdaIN 和 confidence fusion head 来解决室内深度图中大量连续缺失数据的深度完成问题。

室内深度填充的 RGB - 深度融合 CycleGAN (RDFC-GAN)

RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion

While radar and video data can be readily fused at the detection level,
fusing them at the pixel level is potentially more beneficial. This is also
more challenging in part due to the sparsity of radar, but also because
automotive radar beams are much wider than a typical pixel combined with a
large baseline between camera and radar, which results in poor association
between radar pixels and color pixel. A consequence is that depth completion
methods designed for LiDAR and video fare poorly for radar and video. Here we
propose a radar-to-pixel association stage which learns a mapping from radar
returns to pixels. This mapping also serves to densify radar returns. Using
this as a first stage, followed by a more traditional depth completion method,
we are able to achieve image-guided depth completion with radar and video. We
demonstrate performance superior to camera and radar alone on the nuScenes
dataset. Our source code is available at this https URL

该研究探讨了雷达和视频数据的像素级融合，并提出了一种从雷达返回数据到像素的映射，从而在雷达更稀疏的情况下提高车载系统深度感知能力。实验结果表明，在 nuScenes 数据集上，该方法在图像引导深度完成方面表现优于单独使用相机和雷达。