Transformer, as an alternative to CNN, has been proven effective in many
modalities (e.g., texts and images). For 3D point cloud transformers, existing
efforts focus primarily on pushing their accuracy to the state-of-the-art
level. However, their latency lags behind sparse convolution-based models (3x
slower), hindering their usage in resource-constrained, latency-sensitive
applications (such as autonomous driving). This inefficiency comes from point
clouds' sparse and irregular nature, whereas transformers are designed for
dense, regular workloads. This paper presents FlatFormer to close this latency
gap by trading spatial proximity for better computational regularity. We first
flatten the point cloud with window-based sorting and partition points into
groups of equal sizes rather than windows of equal shapes. This effectively
avoids expensive structuring and padding overheads. We then apply
self-attention within groups to extract local features, alternate sorting axis
to gather features from different directions, and shift windows to exchange
features across groups. FlatFormer delivers state-of-the-art accuracy on Waymo
Open Dataset with 4.6x speedup over (transformer-based) SST and 1.4x speedup
over (sparse convolutional) CenterPoint. This is the first point cloud
transformer that achieves real-time performance on edge GPUs and is faster than
sparse convolutional methods while achieving on-par or even superior accuracy
on large-scale benchmarks.

FlatFormer 是一种 3D 点云变压器网络，通过交换计算负荷和提取局部特征以实现节省计算时间的效果，在边缘 GPU 上实现了实时性能，并在大规模基准测试上达到与或甚至更好的准确性。