Multi-modal human action segmentation is a critical and challenging task with
a wide range of applications. Nowadays, the majority of approaches concentrate
on the fusion of dense signals (i.e., RGB, optical flow, and depth maps).
However, the potential contributions of sparse IoT sensor signals, which can be
crucial for achieving accurate recognition, have not been fully explored. To
make up for this, we introduce a Sparse signalguided Transformer (SigFormer) to
combine both dense and sparse signals. We employ mask attention to fuse
localized features by constraining cross-attention within the regions where
sparse signals are valid. However, since sparse signals are discrete, they lack
sufficient information about the temporal action boundaries. Therefore, in
SigFormer, we propose to emphasize the boundary information at two stages to
alleviate this problem. In the first feature extraction stage, we introduce an
intermediate bottleneck module to jointly learn both category and boundary
features of each dense modality through the inner loss functions. After the
fusion of dense modalities and sparse signals, we then devise a two-branch
architecture that explicitly models the interrelationship between action
category and temporal boundary. Experimental results demonstrate that SigFormer
outperforms the state-of-the-art approaches on a multi-modal action
segmentation dataset from real industrial environments, reaching an outstanding
F1 score of 0.958. The codes and pre-trained models have been available at
this https URL

介绍了一种稀疏信号引导的变压器模型 (SigFormer)，该模型通过融合密集信号和稀疏信号来提高多模态人体动作分割的准确性，在处理稀疏信号时使用了蒙版注意力机制，同时在提取特征和建模动作类别与时间边界关系时强调了边界信息。在实验中，SigFormer 在真实工业环境下的多模态动作分割数据集上表现优秀，F1 得分达到 0.958。