Multimodal-based action recognition methods have achieved high success using
pose and RGB modality. However, skeletons sequences lack appearance depiction
and RGB images suffer irrelevant noise due to modality limitations. To address
this, we introduce human parsing feature map as a novel modality, since it can
selectively retain effective semantic features of the body parts, while
filtering out most irrelevant noise. We propose a new dual-branch framework
called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to
leverage both skeletons and human parsing modalities for action recognition.
The first human pose branch feeds robust skeletons in graph convolutional
network to model pose features, while the second human parsing branch also
leverages depictive parsing feature maps to model parsing festures via
convolutional backbones. The two high-level features will be effectively
combined through a late fusion strategy for better action recognition.
Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently
verify the effectiveness of our proposed EPP-Net, which outperforms the
existing action recognition methods. Our code is available at:
this https URL

采用人体解析特征图作为一种新的模态，结合骨架和人体解析模态，提出了一种新的双分支框架（EPP-Net），通过后期融合策略有效地结合两种高级特征以实现更好的动作识别。

探索人工解析模式用于动作识别

Explore Human Parsing Modality for Action Recognition

Human skeletons and RGB sequences are both widely-adopted input modalities
for human action recognition. However, skeletons lack appearance features and
color data suffer large amount of irrelevant depiction. To address this, we
introduce human parsing feature map as a novel modality, since it can
selectively retain spatiotemporal features of the body parts, while filtering
out noises regarding outfits, backgrounds, etc. We propose an Integrating Human
Parsing and Pose Network (IPP-Net) for action recognition, which is the first
to leverage both skeletons and human parsing feature maps in dual-branch
approach. The human pose branch feeds compact skeletal representations of
different modalities in graph convolutional network to model pose features. In
human parsing branch, multi-frame body-part parsing features are extracted with
human detector and parser, which is later learnt using a convolutional
backbone. A late ensemble of two branches is adopted to get final predictions,
considering both robust keypoints and rich semantic body-part features.
Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently
verify the effectiveness of the proposed IPP-Net, which outperforms the
existing action recognition methods. Our code is publicly available at
this https URL .

该研究介绍了一种新的人体解析特征图，被提出用于人体动作识别，结合 IPP-Net 方法，将骨架和人体解析特征映射应用于双分支方法，以筛选人体空间和时间特征，设计出包含人体姿态和人体解析分支的网络体系结构，通过多个数据集的综合对比实验，证明了这种方法的优越性能。