Action recognition has been a heated topic in computer vision for its wide
application in vision systems. Previous approaches achieve improvement by
fusing the modalities of the skeleton sequence and RGB video. However, such
methods have a dilemma between the accuracy and efficiency for the high
complexity of the RGB video network. To solve the problem, we propose a
multi-modality feature fusion network to combine the modalities of the skeleton
sequence and RGB frame instead of the RGB video, as the key information
contained by the combination of skeleton sequence and RGB frame is close to
that of the skeleton sequence and RGB video. In this way, the complementary
information is retained while the complexity is reduced by a large margin. To
better explore the correspondence of the two modalities, a two-stage fusion
framework is introduced in the network. In the early fusion stage, we introduce
a skeleton attention module that projects the skeleton sequence on the single
RGB frame to help the RGB frame focus on the limb movement regions. In the late
fusion stage, we propose a cross-attention module to fuse the skeleton feature
and the RGB feature by exploiting the correlation. Experiments on two
benchmarks NTU RGB+D and SYSU show that the proposed model achieves competitive
performance compared with the state-of-the-art methods while reduces the
complexity of the network.

本文研究动作识别，提出一种基于多模态特征融合网络的方法，将骨架序列和 RGB 帧的关键信息相结合，以此保留补充信息的同时大大降低了其网络的复杂性。实验表明，所提出的模型能够与现有最先进的方法相比取得竞争性的表现。