Transformers have gained much attention by outperforming convolutional neural
networks in many 2D vision tasks. However, they are known to have
generalization problems and rely on massive-scale pre-training and
sophisticated training techniques. When applying to 3D tasks, the irregular
data structure and limited data scale add to the difficulty of transformer's
application. We propose CodedVTR (Codebook-based Voxel TRansformer), which
improves data efficiency and generalization ability for 3D sparse voxel
transformers. On the one hand, we propose the codebook-based attention that
projects an attention space into its subspace represented by the combination of
"prototypes" in a learnable codebook. It regularizes attention learning and
improves generalization. On the other hand, we propose geometry-aware
self-attention that utilizes geometric information (geometric pattern, density)
to guide attention learning. CodedVTR could be embedded into existing sparse
convolution-based methods, and bring consistent performance improvements for
indoor and outdoor 3D semantic segmentation tasks

提出基于码本的体素 Transformer（CodedVTR）以提高 3D 稀疏体素 Transformer 的数据效率和泛化能力，其中包括通过学习可学习码本中的原型来规范化注意力学习和改善泛化能力，并利用几何信息（几何模式、密度）指导注意力学习。CodedVTR 可嵌入现有的基于稀疏卷积的方法，并为室内和室外 3D 语义分割任务带来一致的性能提升。

CodedVTR：基于码本的稀疏体素变换器以及几何引导

CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric  Guidance

We present a self-supervised task on point clouds, in order to learn
meaningful point-wise features that encode local structure around each point.
Our self-supervised network, named MortonNet, operates directly on
unstructured/unordered point clouds. Using a multi-layer RNN, MortonNet
predicts the next point in a point sequence created by a popular and fast Space
Filling Curve, the Morton-order curve. The final RNN state (coined Morton
feature) is versatile and can be used in generic 3D tasks on point clouds. In
fact, we show how Morton features can be used to significantly improve
performance (+3% for 2 popular semantic segmentation algorithms) in the task of
semantic segmentation of point clouds on the challenging and large-scale S3DIS
dataset. We also show how MortonNet trained on S3DIS transfers well to another
large-scale dataset, vKITTI, leading to an improvement over state-of-the-art of
3.8%. Finally, we use Morton features to train a much simpler and more stable
model for part segmentation in ShapeNet. Our results show how our
self-supervised task results in features that are useful for 3D segmentation
tasks, and generalize well to other datasets.

该论文提出了一种自我监督的点云任务，使用 MortonNet 网络直接在无序的点云上进行操作，预测序列中下一个点，生成的 Morton feature 对于 3D 分割等任务具有很好的性能，并在不同的数据集上都有良好的泛化性能。