Depth information has proven to be a useful cue in the semantic segmentation
of RGB-D images for providing a geometric counterpart to the RGB
representation. Most existing works simply assume that depth measurements are
accurate and well-aligned with the RGB pixels and models the problem as a
cross-modal feature fusion to obtain better feature representations to achieve
more accurate segmentation. This, however, may not lead to satisfactory results
as actual depth data are generally noisy, which might worsen the accuracy as
the networks go deeper.
In this paper, we propose a unified and efficient Cross-modality Guided
Encoder to not only effectively recalibrate RGB feature responses, but also to
distill accurate depth information via multiple stages and aggregate the two
recalibrated representations alternatively. The key of the proposed
architecture is a novel Separation-and-Aggregation Gating operation that
jointly filters and recalibrates both representations before cross-modality
aggregation. Meanwhile, a Bi-direction Multi-step Propagation strategy is
introduced, on the one hand, to help to propagate and fuse information between
the two modalities, and on the other hand, to preserve their specificity along
the long-term propagation process. Besides, our proposed encoder can be easily
injected into the previous encoder-decoder structures to boost their
performance on RGB-D semantic segmentation. Our model outperforms
state-of-the-arts consistently on both in-door and out-door challenging
datasets. Code of this work is available at this https URL

本文提出了一种统一且高效的跨模态引导编码器，通过分离 - 聚合门技术，不仅可以有效地校准 RGB 特征响应，而且可以通过多个阶段和交替聚合两个重新校准的表示来提取准确的深度信息，同时引入双向多步传播策略来提高 RGB-D 语义分割性能.