Multimodal-based action recognition methods have achieved high success using
pose and RGB modality. However, skeletons sequences lack appearance depiction
and RGB images suffer irrelevant noise due to modality limitations. To address
this, we introduce human parsing feature map as a novel modality, since it can
selectively retain effective semantic features of the body parts, while
filtering out most irrelevant noise. We propose a new dual-branch framework
called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to
leverage both skeletons and human parsing modalities for action recognition.
The first human pose branch feeds robust skeletons in graph convolutional
network to model pose features, while the second human parsing branch also
leverages depictive parsing feature maps to model parsing festures via
convolutional backbones. The two high-level features will be effectively
combined through a late fusion strategy for better action recognition.
Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently
verify the effectiveness of our proposed EPP-Net, which outperforms the
existing action recognition methods. Our code is available at:
this https URL

采用人体解析特征图作为一种新的模态，结合骨架和人体解析模态，提出了一种新的双分支框架（EPP-Net），通过后期融合策略有效地结合两种高级特征以实现更好的动作识别。