3D spatial information is known to be beneficial to the semantic segmentation
task. Most existing methods take 3D spatial data as an additional input,
leading to a two-stream segmentation network that processes RGB and 3D spatial
information separately. This solution greatly increases the inference time and
severely limits its scope for real-time applications. To solve this problem, we
propose Spatial information guided Convolution (S-Conv), which allows efficient
RGB feature and 3D spatial information integration. S-Conv is competent to
infer the sampling offset of the convolution kernel guided by the 3D spatial
information, helping the convolutional layer adjust the receptive field and
adapt to geometric transformations. S-Conv also incorporates geometric
information into the feature learning process by generating spatially adaptive
convolutional weights. The capability of perceiving geometry is largely
enhanced without much affecting the amount of parameters and computational
cost. We further embed S-Conv into a semantic segmentation network, called
Spatial information Guided convolutional Network (SGNet), resulting in
real-time inference and state-of-the-art performance on NYUDv2 and SUNRGBD
datasets.

提出 Spatial information guided Convolution（简称 S-Conv）算法，将 RGB 特征与 3D 空间信息高效地集成，不会严重影响实时应用，可用于语义分割任务，实现出色的性能，并在 NYUDv2 和 SUNRGBD 数据集上获得最先进的性能。