We present High-Density Visual Particle Dynamics (HD-VPD), a learned world
model that can emulate the physical dynamics of real scenes by processing
massive latent point clouds containing 100K+ particles. To enable efficiency at
this scale, we introduce a novel family of Point Cloud Transformers (PCTs)
called Interlacers leveraging intertwined linear-attention Performer layers and
graph-based neighbour attention layers. We demonstrate the capabilities of
HD-VPD by modeling the dynamics of high degree-of-freedom bi-manual robots with
two RGB-D cameras. Compared to the previous graph neural network approach, our
Interlacer dynamics is twice as fast with the same prediction quality, and can
achieve higher quality using 4x as many particles. We illustrate how HD-VPD can
evaluate motion plan quality with robotic box pushing and can grasping tasks.
See videos and particle dynamics rendered by HD-VPD at
this https URL

通过处理包含 10 万 + 粒子的大规模潜在点云，我们提出了一个学习的世界模型 —— 高密度视觉粒子动力学（HD-VPD），它可以模拟真实场景的物理动力学。为了在这一规模上实现高效率，我们引入了一种新颖的点云变换器（PCTs）家族，称为 Interlacer，利用交织的线性注意力 Performer 层和基于图的邻居注意力层。相比之前的图神经网络方法，我们的 Interlacer 动力学速度是原来的两倍，预测质量相同，并且使用 4 倍的粒子可以实现更高的质量。我们展示了 HD-VPD 在建模具有高自由度的双手机器人与两个 RGB-D 相机的动力学方面的能力。我们演示了 HD-VPD 如何通过机器人推箱和抓取任务来评估运动计划的质量。请在此 https URL 上观看通过 HD-VPD 渲染的视频和粒子动力学。

用高密度视觉粒子动力学模拟现实世界

Modeling the Real World with High-Density Visual Particle Dynamics

The booming of 3D recognition in the 2020s began with the introduction of
point cloud transformers. They quickly overwhelmed sparse CNNs and became
state-of-the-art models, especially in 3D semantic segmentation. However,
sparse CNNs are still valuable networks, due to their efficiency treasure, and
ease of application. In this work, we reexamine the design distinctions and
test the limits of what a sparse CNN can achieve. We discover that the key
credit to the performance difference is adaptivity. Specifically, we propose
two key components, i.e., adaptive receptive fields (spatially) and adaptive
relation, to bridge the gap. This exploration led to the creation of
Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a
lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal
computational cost. Without any self-attention modules, OA-CNNs favorably
surpass point transformers in terms of accuracy in both indoor and outdoor
scenes, with much less latency and memory cost. Notably, it achieves 76.1%,
78.9%, and 70.6% mIoU on ScanNet v2, nuScenes, and SemanticKITTI validation
benchmarks respectively, while maintaining at most 5x better speed than
transformer counterparts. This revelation highlights the potential of pure
sparse CNNs to outperform transformer-related networks.

通过提出自适应感受野和自适应关系两个核心组件，我们发现在保持计算成本最小的前提下，纯稀疏卷积神经网络在室内外场景中的准确性方面优于点云变换网络，且速度至少有 5 倍之多的提升。这一发现突显了纯稀疏卷积神经网络超越变换网络的潜力。

OA-CNNs: 三维语义分割的全能自适应稀疏卷积神经网络

OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation

The recent surge in 3D data acquisition has spurred the development of
geometric deep learning models for point cloud processing, boosted by the
remarkable success of transformers in natural language processing. While point
cloud transformers (PTs) have achieved impressive results recently, their
quadratic scaling with respect to the point cloud size poses a significant
scalability challenge for real-world applications. To address this issue, we
propose the Adaptive Point Cloud Transformer (AdaPT), a standard PT model
augmented by an adaptive token selection mechanism. AdaPT dynamically reduces
the number of tokens during inference, enabling efficient processing of large
point clouds. Furthermore, we introduce a budget mechanism to flexibly adjust
the computational cost of the model at inference time without the need for
retraining or fine-tuning separate models. Our extensive experimental
evaluation on point cloud classification tasks demonstrates that AdaPT
significantly reduces computational complexity while maintaining competitive
accuracy compared to standard PTs. The code for AdaPT is made publicly
available.

通过提出自适应点云变换器（AdaPT）模型，文章解决了点云变换器（PT）在处理大规模点云时的可扩展性挑战，实现了较低的计算复杂度和竞争性的分类准确率。