We present DFormer, a novel RGB-D pretraining framework to learn transferable
representations for RGB-D segmentation tasks. DFormer has two new key
innovations: 1) Unlike previous works that aim to encode RGB features,DFormer
comprises a sequence of RGB-D blocks, which are tailored for encoding both RGB
and depth information through a novel building block design; 2) We pre-train
the backbone using image-depth pairs from ImageNet-1K, and thus the DFormer is
endowed with the capacity to encode RGB-D representations. It avoids the
mismatched encoding of the 3D geometry relationships in depth maps by RGB
pre-trained backbones, which widely lies in existing methods but has not been
resolved. We fine-tune the pre-trained DFormer on two popular RGB-D tasks,
i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a
lightweight decoder head. Experimental results show that our DFormer achieves
new state-of-the-art performance on these two tasks with less than half of the
computational cost of the current best methods on two RGB-D segmentation
datasets and five RGB-D saliency datasets. Our code is available at:
this https URL

DFormer 是一种创新的 RGB-D 预训练框架，通过使用一系列 RGB-D 块进行编码，将 RGB 和深度信息编码成可转移的表示，并避免了现有方法中 RGB 预训练的主干网络对深度图中的三维几何关系进行不匹配的编码问题。在两个常见的 RGB-D 任务上，使用轻量级的解码器头微调预训练的 DFormer，该方法在两个 RGB-D 分割数据集和五个 RGB-D 显着性数据集上以不到当前最佳方法计算成本一半的代价实现了最新的最佳性能。

DFormer：重新思考语义分割的 RGBD 表示学习

DFormer: Rethinking RGBD Representation Learning for Semantic  Segmentation

Analyzing scenes thoroughly is crucial for mobile robots acting in different
environments. Semantic segmentation can enhance various subsequent tasks, such
as (semantically assisted) person perception, (semantic) free space detection,
(semantic) mapping, and (semantic) navigation. In this paper, we propose an
efficient and robust RGB-D segmentation approach that can be optimized to a
high degree using NVIDIA TensorRT and, thus, is well suited as a common initial
processing step in a complex system for scene analysis on mobile robots. We
show that RGB-D segmentation is superior to processing RGB images solely and
that it can still be performed in real time if the network architecture is
carefully designed. We evaluate our proposed Efficient Scene Analysis Network
(ESANet) on the common indoor datasets NYUv2 and SUNRGB-D and show that we
reach state-of-the-art performance while enabling faster inference.
Furthermore, our evaluation on the outdoor dataset Cityscapes shows that our
approach is suitable for other areas of application as well. Finally, instead
of presenting benchmark results only, we also show qualitative results in one
of our indoor application scenarios.

本文介绍了一种使用 NVIDIA TensorRT 进行优化的高效且稳健的 RGB-D 分割方法，该方法可作为场景分析系统的公共初始处理步骤。我们通过使用 ESANet 在 NYUv2 和 SUNRGB-D 室内数据集上进行评估，展示了 RGB-D 分割优于仅处理 RGB 图像的效果，并证明当网络架构精心设计时仍可实时完成。此外，我们在 Cityscapes 的室外数据集上进行评估，显示了我们的方法也适用于其他应用领域，并展示了我们在一个室内应用场景中的定性结果.