4D panoptic segmentation is a challenging but practically useful task that
requires every point in a LiDAR point-cloud sequence to be assigned a semantic
class label, and individual objects to be segmented and tracked over time.
Existing approaches utilize only LiDAR inputs which convey limited information
in regions with point sparsity. This problem can, however, be mitigated by
utilizing RGB camera images which offer appearance-based information that can
reinforce the geometry-based LiDAR features. Motivated by this, we propose
4D-Former: a novel method for 4D panoptic segmentation which leverages both
LiDAR and image modalities, and predicts semantic masks as well as temporally
consistent object masks for the input point-cloud sequence. We encode semantic
classes and objects using a set of concise queries which absorb feature
information from both data modalities. Additionally, we propose a learned
mechanism to associate object tracks over time which reasons over both
appearance and spatial location. We apply 4D-Former to the nuScenes and
SemanticKITTI datasets where it achieves state-of-the-art results.

通过整合 LiDAR 和图像数据，4D-Former 方法在 4D 全景分割任务上取得了最先进的结果，能够为 LiDAR 序列中的每个点分配语义类别标签和进行对象分割，并能在时间上保持一致的对象分割遮罩。

4D-Former: 多模态 4D 全景分割

4D-Former: Multimodal 4D Panoptic Segmentation

Accurately perceiving and tracking instances over time is essential for the
decision-making processes of autonomous agents interacting safely in dynamic
environments. With this intention, we propose Mask4D for the challenging task
of 4D panoptic segmentation of LiDAR point clouds. Mask4D is the first
transformer-based approach unifying semantic instance segmentation and tracking
of sparse and irregular sequences of 3D point clouds into a single joint model.
Our model directly predicts semantic instances and their temporal associations
without relying on any hand-crafted non-learned association strategies such as
probabilistic clustering or voting-based center prediction. Instead, Mask4D
introduces spatio-temporal instance queries which encode the semantic and
geometric properties of each semantic tracklet in the sequence. In an in-depth
study, we find that it is critical to promote spatially compact instance
predictions as spatio-temporal instance queries tend to merge multiple
semantically similar instances, even if they are spatially distant. To this
end, we regress 6-DOF bounding box parameters from spatio-temporal instance
queries, which is used as an auxiliary task to foster spatially compact
predictions. Mask4D achieves a new state-of-the-art on the SemanticKITTI test
set with a score of 68.4 LSTQ, improving upon published top-performing methods
by at least +4.5%.

Mask4D 是一种基于 Transformer 的方法，用于将 LiDAR 点云进行 4D 全景分割的挑战性任务，直接预测语义实例及其时间关联，无需借助任何手工设计的非学习关联策略，通过引入时空实例查询编码每个实例轨迹的语义和几何特性，并从时空实例查询回归 6DOF 边界框参数以促进紧凑的空间预测，最终在 SemanticKITTI 测试集上取得了 68.4 LSTQ 的新的最优效果，相较于已发表的表现最佳方法提高了至少 4.5%。