Recently, pose-based action recognition has gained more and more attention
due to the better performance compared with traditional appearance-based
methods. However, there still exist two problems to be further solved. First,
existing pose-based methods generally recognize human actions with captured 3D
human poses which are very difficult to obtain in real scenarios. Second, few
pose-based methods model the action-related objects in recognizing human-object
interaction actions in which objects play an important role. To solve the
problems above, we propose a pose-based two-stream relational network (PSRN)
for action recognition. In PSRN, one stream models the temporal dynamics of the
targeted 2D human pose sequences which are directly extracted from raw videos,
and the other stream models the action-related objects from a randomly sampled
video frame. Most importantly, instead of fusing two-streams in the class score
layer as before, we propose a pose-object relational network to model the
relationship between human poses and action-related objects. We evaluate the
proposed PSRN on two challenging benchmarks, i.e., Sub-JHMDB and PennAction.
Experimental results show that our PSRN obtains the state-the-of-art
performance on Sub-JHMDB (80.2%) and PennAction (98.1%). Our work opens a new
door to action recognition by combining 2D human pose extracted from raw video
and image appearance.

提出了基于姿态的双流关系网络（PSRN）来解决目前基于姿态的动作识别存在的问题，该方法包括两个流，一个流用于建模目标 2D 人体姿势序列的时间动态，另一个流用于建模与动作相关的物体，通过姿态对象关系网络建模了人体姿态和动作相关物体之间的关系，在 Sub-JHMDB 和 PennAction 数据集上取得了最新的性能表现。

基于姿态的双流关系网络用于视频动作识别

Pose-Based Two-Stream Relational Networks for Action Recognition in  Videos

In this work we study the use of 3D hand poses to recognize first-person
dynamic hand actions interacting with 3D objects. Towards this goal, we
collected RGB-D video sequences comprised of more than 100K frames of 45 daily
hand action categories, involving 26 different objects in several hand
configurations. To obtain hand pose annotations, we used our own mo-cap system
that automatically infers the 3D location of each of the 21 joints of a hand
model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded
the 6D object poses and provide 3D object models for a subset of hand-object
interaction sequences. To the best of our knowledge, this is the first
benchmark that enables the study of first-person hand actions with the use of
3D hand poses. We present an extensive experimental evaluation of RGB-D and
pose-based action recognition by 18 baselines/state-of-the-art approaches. The
impact of using appearance features, poses, and their combinations are
measured, and the different training/testing protocols are evaluated. Finally,
we assess how ready the 3D hand pose estimation field is when hands are
severely occluded by objects in egocentric views and its influence on action
recognition. From the results, we see clear benefits of using hand pose as a
cue for action recognition compared to other data modalities. Our dataset and
experiments can be of interest to communities of 3D hand pose estimation, 6D
object pose, and robotics as well as action recognition.

本文提出了一种基于 3D 手部姿势的第一人称动态手部动作识别方法，并通过收集大量 RGB-D 视频序列对其进行了实验评估。研究结果表明，在进行动作识别时，与其他数据形式相比，手部姿态作为一种线索具有明显的优势。