Understanding 4D point cloud sequences online is of significant practical
value in various scenarios such as VR/AR, robotics, and autonomous driving. The
key goal is to continuously analyze the geometry and dynamics of a 3D scene as
unstructured and redundant point cloud sequences arrive. And the main challenge
is to effectively model the long-term history while keeping computational costs
manageable. To tackle these challenges, we introduce a generic online 4D
perception paradigm called NSM4D. NSM4D serves as a plug-and-play strategy that
can be adapted to existing 4D backbones, significantly enhancing their online
perception capabilities for both indoor and outdoor scenarios. To efficiently
capture the redundant 4D history, we propose a neural scene model that
factorizes geometry and motion information by constructing geometry tokens
separately storing geometry and motion features. Exploiting the history becomes
as straightforward as querying the neural scene model. As the sequence
progresses, the neural scene model dynamically deforms to align with new
observations, effectively providing the historical context and updating itself
with the new observations. By employing token representation, NSM4D also
exhibits robustness to low-level sensor noise and maintains a compact size
through a geometric sampling scheme. We integrate NSM4D with state-of-the-art
4D perception backbones, demonstrating significant improvements on various
online perception benchmarks in indoor and outdoor settings. Notably, we
achieve a 9.6% accuracy improvement for HOI4D online action segmentation and a
3.4% mIoU improvement for SemanticKITTI online semantic segmentation.
Furthermore, we show that NSM4D inherently offers excellent scalability to
longer sequences beyond the training set, which is crucial for real-world
applications.

NSM4D 是一种在线的 4D 感知范式，能够有效处理 4D 点云序列的几何和动态信息，并在室内外场景中实现在线感知能力的显著提升。

NSM4D：基于神经场景模型的在线 4D 点云序列理解

NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence  Understanding

While recent camera-only 3D detection methods leverage multiple timesteps,
the limited history they use significantly hampers the extent to which temporal
fusion can improve object perception. Observing that existing works' fusion of
multi-frame images are instances of temporal stereo matching, we find that
performance is hindered by the interplay between 1) the low granularity of
matching resolution and 2) the sub-optimal multi-view setup produced by limited
history usage. Our theoretical and empirical analysis demonstrates that the
optimal temporal difference between views varies significantly for different
pixels and depths, making it necessary to fuse many timesteps over long-term
history. Building on our investigation, we propose to generate a cost volume
from a long history of image observations, compensating for the coarse but
efficient matching resolution with a more optimal multi-view matching setup.
Further, we augment the per-frame monocular depth predictions used for
long-term, coarse matching with short-term, fine-grained matching and find that
long and short term temporal fusion are highly complementary. While maintaining
high efficiency, our framework sets new state-of-the-art on nuScenes, achieving
first place on the test set and outperforming previous best art by 5.2% mAP and
3.7% NDS on the validation set. Code will be released
$\href{this https URL}{here.}$

该论文介绍了一种新的视频目标检测方法，通过多帧图像数据的长期历史建立成本卷积表示，从而将高效和高质量的匹配相结合，并在 nuScenes 数据集上达到了最新的最佳性能。

时间见证：时态多视角三维物体检测的新前景和基准

Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

Human motion prediction aims to forecast future human poses given a
historical motion. Whether based on recurrent or feed-forward neural networks,
existing learning based methods fail to model the observation that human motion
tends to repeat itself, even for complex sports actions and cooking activities.
Here, we introduce an attention based feed-forward network that explicitly
leverages this observation. In particular, instead of modeling frame-wise
attention via pose similarity, we propose to extract motion attention to
capture the similarity between the current motion context and the historical
motion sub-sequences. In this context, we study the use of different types of
attention, computed at joint, body part, and full pose levels. Aggregating the
relevant past motions and processing the result with a graph convolutional
network allows us to effectively exploit motion patterns from the long-term
history to predict the future poses. Our experiments on Human3.6M, AMASS and
3DPW validate the benefits of our approach for both periodical and
non-periodical actions. Thanks to our attention model, it yields
state-of-the-art results on all three datasets. Our code is available at
this https URL

介绍了一种基于注意力机制的前馈神经网络，利用运动的重复性和历史上下文进行运动预测，通过图卷积网络对历史数据的运动模式进行有效利用，取得了三个数据集上最先进的结果。