Understanding 4D point cloud sequences online is of significant practical
value in various scenarios such as VR/AR, robotics, and autonomous driving. The
key goal is to continuously analyze the geometry and dynamics of a 3D scene as
unstructured and redundant point cloud sequences arrive. And the main challenge
is to effectively model the long-term history while keeping computational costs
manageable. To tackle these challenges, we introduce a generic online 4D
perception paradigm called NSM4D. NSM4D serves as a plug-and-play strategy that
can be adapted to existing 4D backbones, significantly enhancing their online
perception capabilities for both indoor and outdoor scenarios. To efficiently
capture the redundant 4D history, we propose a neural scene model that
factorizes geometry and motion information by constructing geometry tokens
separately storing geometry and motion features. Exploiting the history becomes
as straightforward as querying the neural scene model. As the sequence
progresses, the neural scene model dynamically deforms to align with new
observations, effectively providing the historical context and updating itself
with the new observations. By employing token representation, NSM4D also
exhibits robustness to low-level sensor noise and maintains a compact size
through a geometric sampling scheme. We integrate NSM4D with state-of-the-art
4D perception backbones, demonstrating significant improvements on various
online perception benchmarks in indoor and outdoor settings. Notably, we
achieve a 9.6% accuracy improvement for HOI4D online action segmentation and a
3.4% mIoU improvement for SemanticKITTI online semantic segmentation.
Furthermore, we show that NSM4D inherently offers excellent scalability to
longer sequences beyond the training set, which is crucial for real-world
applications.

NSM4D 是一种在线的 4D 感知范式，能够有效处理 4D 点云序列的几何和动态信息，并在室内外场景中实现在线感知能力的显著提升。