Three dimensional convolutional neural networks (3D CNNs) have been
established as a powerful tool to simultaneously learn features from both
spatial and temporal dimensions, which is suitable to be applied to video-based
action recognition. In this work, we propose not to directly use the
activations of fully-connected layers of a 3D CNN as the video feature, but to
use selective convolutional layer activations to form a discriminative
descriptor for video. It pools the feature on the convolutional layers under
the guidance of body joint positions. Two schemes of mapping body joints into
convolutional feature maps for pooling are discussed. The body joint positions
can be obtained from any off-the-shelf skeleton estimation algorithm. The
helpfulness of the body joint guided feature pooling with inaccurate skeleton
estimation is systematically evaluated. To make it end-to-end and do not rely
on any sophisticated body joint detection algorithm, we further propose a
two-stream bilinear model which can learn the guidance from the body joints and
capture the spatio-temporal features simultaneously. In this model, the body
joint guided feature pooling is conveniently formulated as a bilinear product
operation. Experimental results on three real-world datasets demonstrate the
effectiveness of body joint guided pooling which achieves promising
performance.

该研究提出了一种利用卷积层激活函数形成判别视频描述符的方法，在身体关节位置的指引下对卷积层数上的特征进行聚合，从而提升视频识别的准确性，并使用双线性模型来使其端到端化和不依赖于任何复杂的身体关节检测算法。