Transformers have gained much attention by outperforming convolutional neural
networks in many 2D vision tasks. However, they are known to have
generalization problems and rely on massive-scale pre-training and
sophisticated training techniques. When applying to 3D tasks, the irregular
data structure and limited data scale add to the difficulty of transformer's
application. We propose CodedVTR (Codebook-based Voxel TRansformer), which
improves data efficiency and generalization ability for 3D sparse voxel
transformers. On the one hand, we propose the codebook-based attention that
projects an attention space into its subspace represented by the combination of
"prototypes" in a learnable codebook. It regularizes attention learning and
improves generalization. On the other hand, we propose geometry-aware
self-attention that utilizes geometric information (geometric pattern, density)
to guide attention learning. CodedVTR could be embedded into existing sparse
convolution-based methods, and bring consistent performance improvements for
indoor and outdoor 3D semantic segmentation tasks

提出基于码本的体素 Transformer（CodedVTR）以提高 3D 稀疏体素 Transformer 的数据效率和泛化能力，其中包括通过学习可学习码本中的原型来规范化注意力学习和改善泛化能力，并利用几何信息（几何模式、密度）指导注意力学习。CodedVTR 可嵌入现有的基于稀疏卷积的方法，并为室内和室外 3D 语义分割任务带来一致的性能提升。