RGB-D semantic segmentation has attracted increasing attention over the past
few years. Existing methods mostly employ homogeneous convolution operators to
consume the RGB and depth features, ignoring their intrinsic differences. In
fact, the RGB values capture the photometric appearance properties in the
projected image space, while the depth feature encodes both the shape of a
local geometry as well as the base (whereabout) of it in a larger context.
Compared with the base, the shape probably is more inherent and has a stronger
connection to the semantics, and thus is more critical for segmentation
accuracy. Inspired by this observation, we introduce a Shape-aware
Convolutional layer (ShapeConv) for processing the depth feature, where the
depth feature is firstly decomposed into a shape-component and a
base-component, next two learnable weights are introduced to cooperate with
them independently, and finally a convolution is applied on the re-weighted
combination of these two components. ShapeConv is model-agnostic and can be
easily integrated into most CNNs to replace vanilla convolutional layers for
semantic segmentation. Extensive experiments on three challenging indoor RGB-D
semantic segmentation benchmarks, i.e., NYU-Dv2(-13,-40), SUN RGB-D, and SID,
demonstrate the effectiveness of our ShapeConv when employing it over five
popular architectures. Moreover, the performance of CNNs with ShapeConv is
boosted without introducing any computation and memory increase in the
inference phase. The reason is that the learnt weights for balancing the
importance between the shape and base components in ShapeConv become constants
in the inference phase, and thus can be fused into the following convolution,
resulting in a network that is identical to one with vanilla convolutional
layers.

本文提出了一种新的深度学习卷积方法 ShapeConv，用于处理三维点云数据的语义分割问题，该方法将深度特征分解成形状和基础两个部分进行卷积操作，不仅有效提高了分类的准确性，而且不会增加计算和存储成本。

ShapeConv: 针对室内 RGB-D 语义分割的形状感知卷积层

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic  Segmentation

Point cloud is a principal data structure adopted for 3D geometric
information encoding. Unlike other conventional visual data, such as images and
videos, these irregular points describe the complex shape features of 3D
objects, which makes shape feature learning an essential component of point
cloud analysis. To this end, a shape-oriented message passing scheme dubbed
ShapeConv is proposed to focus on the representation learning of the underlying
shape formed by each local neighboring point. Despite this intra-shape
relationship learning, ShapeConv is also designed to incorporate the contextual
effects from the inter-shape relationship through capturing the long-ranged
dependencies between local underlying shapes. This shape-oriented operator is
stacked into our hierarchical learning architecture, namely Shape-Oriented
Convolutional Neural Network (SOCNN), developed for point cloud analysis.
Extensive experiments have been performed to evaluate its significance in the
tasks of point cloud classification and part segmentation.

本文提出了一种形状取向的消息传递方案 ShapeConv，结合了形状特征学习和上下文信息对基于点云的分析的贡献，并将其堆叠到分层学习体系结构 SOCNN 中以分析点云的分类和分割任务。