Online semantic 3D segmentation in company with real-time RGB-D
reconstruction poses special challenges such as how to perform 3D convolution
directly over the progressively fused 3D geometric data, and how to smartly
fuse information from frame to frame. We propose a novel fusion-aware 3D point
convolution which operates directly on the geometric surface being
reconstructed and exploits effectively the inter-frame correlation for high
quality 3D feature learning. This is enabled by a dedicated dynamic data
structure which organizes the online acquired point cloud with global-local
trees. Globally, we compile the online reconstructed 3D points into an
incrementally growing coordinate interval tree, enabling fast point insertion
and neighborhood query. Locally, we maintain the neighborhood information for
each point using an octree whose construction benefits from the fast query of
the global tree.Both levels of trees update dynamically and help the 3D
convolution effectively exploits the temporal coherence for effective
information fusion across RGB-D frames.

该研究提出了一种基于动态数据结构的卷积方法，通过全局 - 本地树结构维护局部领域信息，以及增量坐标区间树实现快速点云插入和邻域查询，从而解决在线语义三维分割与 RGB-D 重建中的挑战，实现跨帧时间上的信息融合。