Various types of sensors have been considered to develop human action
recognition (HAR) models. Robust HAR performance can be achieved by fusing
multimodal data acquired by different sensors. In this paper, we introduce a
new multimodal fusion architecture, referred to as Unified Contrastive Fusion
Transformer (UCFFormer) designed to integrate data with diverse distributions
to enhance HAR performance. Based on the embedding features extracted from each
modality, UCFFormer employs the Unified Transformer to capture the
inter-dependency among embeddings in both time and modality domains. We present
the Factorized Time-Modality Attention to perform self-attention efficiently
for the Unified Transformer. UCFFormer also incorporates contrastive learning
to reduce the discrepancy in feature distributions across various modalities,
thus generating semantically aligned features for information fusion.
Performance evaluation conducted on two popular datasets, UTD-MHAD and NTU
RGB+D, demonstrates that UCFFormer achieves state-of-the-art performance,
outperforming competing methods by considerable margins.

提出了一种新的多模态融合架构 UCFFormer，它能够整合具有不同分布的数据以增强人类动作识别 (HAR) 的性能；通过使用统一 Transformer 来捕捉嵌入特征在时间和模态领域之间的相互依赖关系，并引入了分解的时间 - 模态注意力来高效执行自注意力，在各种模态之间减少特征分布上的差异，从而生成在语义上对齐的特征进行信息融合；在 UTD-MHAD 和 NTU RGB+D 两个流行数据集上进行的性能评估表明，UCFFormer 通过显著的优势超越竞争方法，实现了最先进的性能。