Pose based hand gesture recognition has been widely studied in the recent
years. Compared with full body action recognition, hand gesture involves joints
that are more spatially closely distributed with stronger collaboration. This
nature requires a different approach from action recognition to capturing the
complex spatial features. Many gesture categories, such as "Grab" and "Pinch",
have very similar motion or temporal patterns posing a challenge on temporal
processing. To address these challenges, this paper proposes a two-stream
neural network with one stream being a self-attention based graph convolutional
network (SAGCN) extracting the short-term temporal information and hierarchical
spatial information, and the other being a residual-connection enhanced
bidirectional Independently Recurrent Neural Network (RBi-IndRNN) for
extracting long-term temporal information. The self-attention based graph
convolutional network has a dynamic self-attention mechanism to adaptively
exploit the relationships of all hand joints in addition to the fixed topology
and local feature extraction in the GCN. On the other hand, the
residual-connection enhanced Bi-IndRNN extends an IndRNN with the capability of
bidirectional processing for temporal modelling. The two streams are fused
together for recognition. The Dynamic Hand Gesture dataset and First-Person
Hand Action dataset are used to validate its effectiveness, and our method
achieves state-of-the-art performance.

本研究提出了基于自注意力图卷积网络和 RBI-IndRNN 的双流神经网络，用于提取短期和长期时间信息，以实现手势识别。在 Dynamic Hand Gesture 数据集和 First-Person Hand Action 数据集上验证了其有效性，并取得了最新的性能。

基于姿态的手势识别的双流神经网络

A Two-stream Neural Network for Pose-based Hand Gesture Recognition

Because of the rich dynamical structure of videos and their ubiquity in
everyday life, it is a natural idea that video data could serve as a powerful
unsupervised learning signal for training visual representations in deep neural
networks. However, instantiating this idea, especially at large scale, has
remained a significant artificial intelligence challenge. Here we present the
Video Instance Embedding (VIE) framework, which extends powerful recent
unsupervised loss functions for learning deep nonlinear embeddings to
multi-stream temporal processing architectures on large-scale video datasets.
We show that VIE-trained networks substantially advance the state of the art in
unsupervised learning from video datastreams, both for action recognition in
the Kinetics dataset, and object recognition in the ImageNet dataset. We show
that a hybrid model with both static and dynamic processing pathways is optimal
for both transfer tasks, and provide analyses indicating how the pathways
differ. Taken in context, our results suggest that deep neural embeddings are a
promising approach to unsupervised visual learning across a wide variety of
domains.

本文介绍了 Video Instance Embedding（VIE）框架，它扩展了用于学习深度非线性嵌入的强大无监督损失函数以进行大规模视频数据集上的多流时间处理架构，展示了 VIE 训练的网络在 Kinetics 数据集的动作识别和 ImageNet 数据集的目标识别中有重大发展，并提供了分析表明路径如何有所不同。