Graph convolutional networks (GCNs), which generalize CNNs to more generic
non-Euclidean structures, have achieved remarkable performance for
skeleton-based action recognition. However, there still exist several issues in
the previous GCN-based models. First, the topology of the graph is set
heuristically and fixed over all the model layers and input data. This may not
be suitable for the hierarchy of the GCN model and the diversity of the data in
action recognition tasks. Second, the second-order information of the skeleton
data, i.e., the length and orientation of the bones, is rarely investigated,
which is naturally more informative and discriminative for the human action
recognition. In this work, we propose a novel multi-stream attention-enhanced
adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based
action recognition. The graph topology in our model can be either uniformly or
individually learned based on the input data in an end-to-end manner. This
data-driven approach increases the flexibility of the model for graph
construction and brings more generality to adapt to various data samples.
Besides, the proposed adaptive graph convolutional layer is further enhanced by
a spatial-temporal-channel attention module, which helps the model pay more
attention to important joints, frames and features. Moreover, the information
of both the joints and bones, together with their motion information, are
simultaneously modeled in a multi-stream framework, which shows notable
improvement for the recognition accuracy. Extensive experiments on the two
large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the
performance of our model exceeds the state-of-the-art with a significant
margin.

本文提出了一种多流注意力增强的自适应图卷积神经网络 (MS-AAGCN)，用于基于骨架的动作识别。该模型可以端到端地学习图的拓扑结构，并用空时通道注意模块提高模型注意到重要关节、帧和特征。多流框架同时建模关节和骨骼信息及其运动信息，结果在大规模数据集 NTU-RGBD 和 Kinetics-Skeleton 上都超过了现有技术水平。

基于骨架的多流自适应图卷积网络动作识别

Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph  Convolutional Networks

This paper studies deep network architectures to address the problem of video
classification. A multi-stream framework is proposed to fully utilize the rich
multimodal information in videos. Specifically, we first train three
Convolutional Neural Networks to model spatial, short-term motion and audio
clues respectively. Long Short Term Memory networks are then adopted to explore
long-term temporal dynamics. With the outputs of the individual streams, we
propose a simple and effective fusion method to generate the final predictions,
where the optimal fusion weights are learned adaptively for each class, and the
learning process is regularized by automatically estimated class relationships.
Our contributions are two-fold. First, the proposed multi-stream framework is
able to exploit multimodal features that are more comprehensive than those
previously attempted. Second, we demonstrate that the adaptive fusion method
using the class relationship as a regularizer outperforms traditional
alternatives that estimate the weights in a "free" fashion. Our framework
produces significantly better results than the state of the arts on two popular
benchmarks, 92.2\% on UCF-101 (without using audio) and 84.9\% on Columbia
Consumer Videos.

本研究提出了一种多流深度网络架构，用于解决视频分类问题。通过训练三种卷积神经网络来建模视频中的空间、短期动态和音频线索，并采用长短期记忆网络来探索长期时间动态，最终通过自适应融合方法生成预测结果，将多模态信息充分利用。实验证明，该方法的性能明显优于现有的方法。