3D object detection has seen quick progress thanks to advances in deep
learning on point clouds. A few recent works have even shown state-of-the-art
performance with just point clouds input (e.g. VoteNet). However, point cloud
data have inherent limitations. They are sparse, lack color information and
often suffer from sensor noise. Images, on the other hand, have high resolution
and rich texture. Thus they can complement the 3D geometry provided by point
clouds. Yet how to effectively use image information to assist point cloud
based detection is still an open question. In this work, we build on top of
VoteNet and propose a 3D detection architecture called ImVoteNet specialized
for RGB-D scenes. ImVoteNet is based on fusing 2D votes in images and 3D votes
in point clouds. Compared to prior work on multi-modal detection, we explicitly
extract both geometric and semantic features from the 2D images. We leverage
camera parameters to lift these features to 3D. To improve the synergy of 2D-3D
feature fusion, we also propose a multi-tower training scheme. We validate our
model on the challenging SUN RGB-D dataset, advancing state-of-the-art results
by 5.7 mAP. We also provide rich ablation studies to analyze the contribution
of each design choice.

本文提出了一种新的三维检测方法 ImVoteNet，通过将二维图像和三维点云进行融合来实现，验证结果表明，此方法在 SUN RGB-D 数据集上优于先前方法，主要应用于 RGB-D 场景。

ImVoteNet: 通过图像投票提升点云中的 3D 物体检测

ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes

Semantic labeling of RGB-D scenes is crucial to many intelligent applications
including perceptual robotics. It generates pixelwise and fine-grained label
maps from simultaneously sensed photometric (RGB) and depth channels. This
paper addresses this problem by i) developing a novel Long Short-Term Memorized
Context Fusion (LSTM-CF) Model that captures and fuses contextual information
from multiple channels of photometric and depth data, and ii) incorporating
this model into deep convolutional neural networks (CNNs) for end-to-end
training. Specifically, contexts in photometric and depth channels are,
respectively, captured by stacking several convolutional layers and a long
short-term memory layer; the memory layer encodes both short-range and
long-range spatial dependencies in an image along the vertical direction.
Another long short-term memorized fusion layer is set up to integrate the
contexts along the vertical direction from different channels, and perform
bi-directional propagation of the fused vertical contexts along the horizontal
direction to obtain true 2D global contexts. At last, the fused contextual
representation is concatenated with the convolutional features extracted from
the photometric channels in order to improve the accuracy of fine-scale
semantic labeling. Our proposed model has set a new state of the art, i.e.,
48.1% and 49.4% average class accuracy over 37 categories (2.2% and 5.4%
improvement) on the large-scale SUNRGBD dataset and the NYUDv2dataset,
respectively.

本文开发了一种新型的 LSTM-CF 模型，它可以从多个光度和深度数据通道捕获和融合上下文信息，并将此模型纳入深度卷积神经网络（CNNs），以用于端到端训练，以提高细粒度语义标签的准确性。