We present DFormer, a novel RGB-D pretraining framework to learn transferable
representations for RGB-D segmentation tasks. DFormer has two new key
innovations: 1) Unlike previous works that aim to encode RGB features,DFormer
comprises a sequence of RGB-D blocks, which are tailored for encoding both RGB
and depth information through a novel building block design; 2) We pre-train
the backbone using image-depth pairs from ImageNet-1K, and thus the DFormer is
endowed with the capacity to encode RGB-D representations. It avoids the
mismatched encoding of the 3D geometry relationships in depth maps by RGB
pre-trained backbones, which widely lies in existing methods but has not been
resolved. We fine-tune the pre-trained DFormer on two popular RGB-D tasks,
i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a
lightweight decoder head. Experimental results show that our DFormer achieves
new state-of-the-art performance on these two tasks with less than half of the
computational cost of the current best methods on two RGB-D segmentation
datasets and five RGB-D saliency datasets. Our code is available at:
this https URL

DFormer 是一种创新的 RGB-D 预训练框架，通过使用一系列 RGB-D 块进行编码，将 RGB 和深度信息编码成可转移的表示，并避免了现有方法中 RGB 预训练的主干网络对深度图中的三维几何关系进行不匹配的编码问题。在两个常见的 RGB-D 任务上，使用轻量级的解码器头微调预训练的 DFormer，该方法在两个 RGB-D 分割数据集和五个 RGB-D 显着性数据集上以不到当前最佳方法计算成本一半的代价实现了最新的最佳性能。

DFormer：重新思考语义分割的 RGBD 表示学习

DFormer: Rethinking RGBD Representation Learning for Semantic  Segmentation

This paper introduces an approach, named DFormer, for universal image
segmentation. The proposed DFormer views universal image segmentation task as a
denoising process using a diffusion model. DFormer first adds various levels of
Gaussian noise to ground-truth masks, and then learns a model to predict
denoising masks from corrupted masks. Specifically, we take deep pixel-level
features along with the noisy masks as inputs to generate mask features and
attention masks, employing diffusion-based decoder to perform mask prediction
gradually. At inference, our DFormer directly predicts the masks and
corresponding categories from a set of randomly-generated masks. Extensive
experiments reveal the merits of our proposed contributions on different image
segmentation tasks: panoptic segmentation, instance segmentation, and semantic
segmentation. Our DFormer outperforms the recent diffusion-based panoptic
segmentation method Pix2Seq-D with a gain of 3.6% on MS COCO val2017 set.
Further, DFormer achieves promising semantic segmentation performance
outperforming the recent diffusion-based method by 2.2% on ADE20K val set. Our
source code and models will be publicly on this https URL

本文介绍了一种名为 DFormer 的方法，用于实现全局图像分割，该方法将全局图像分割任务视为使用扩散模型的降噪过程，在地面真实掩模上添加各个层次的高斯噪声，然后学习模型从损坏掩模中预测降噪掩模，最后使用基于扩散的解码器逐渐执行掩模预测，并直接从一组随机生成的蒙版中预测掩模和相应类别。