RGB-D semantic segmentation has attracted increasing attention over the past
few years. Existing methods mostly employ homogeneous convolution operators to
consume the RGB and depth features, ignoring their intrinsic differences. In
fact, the RGB values capture the photometric appearance properties in the
projected image space, while the depth feature encodes both the shape of a
local geometry as well as the base (whereabout) of it in a larger context.
Compared with the base, the shape probably is more inherent and has a stronger
connection to the semantics, and thus is more critical for segmentation
accuracy. Inspired by this observation, we introduce a Shape-aware
Convolutional layer (ShapeConv) for processing the depth feature, where the
depth feature is firstly decomposed into a shape-component and a
base-component, next two learnable weights are introduced to cooperate with
them independently, and finally a convolution is applied on the re-weighted
combination of these two components. ShapeConv is model-agnostic and can be
easily integrated into most CNNs to replace vanilla convolutional layers for
semantic segmentation. Extensive experiments on three challenging indoor RGB-D
semantic segmentation benchmarks, i.e., NYU-Dv2(-13,-40), SUN RGB-D, and SID,
demonstrate the effectiveness of our ShapeConv when employing it over five
popular architectures. Moreover, the performance of CNNs with ShapeConv is
boosted without introducing any computation and memory increase in the
inference phase. The reason is that the learnt weights for balancing the
importance between the shape and base components in ShapeConv become constants
in the inference phase, and thus can be fused into the following convolution,
resulting in a network that is identical to one with vanilla convolutional
layers.

本文提出了一种新的深度学习卷积方法 ShapeConv，用于处理三维点云数据的语义分割问题，该方法将深度特征分解成形状和基础两个部分进行卷积操作，不仅有效提高了分类的准确性，而且不会增加计算和存储成本。