This paper proposes three simple, compact yet effective representations of
depth sequences, referred to respectively as Dynamic Depth Images (DDI),
Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images
(DDMNI), for both isolated and continuous action recognition. These dynamic
images are constructed from a segmented sequence of depth maps using
hierarchical bidirectional rank pooling to effectively capture the
spatial-temporal information. Specifically, DDI exploits the dynamics of
postures over time and DDNI and DDMNI exploit the 3D structural information
captured by depth maps. Upon the proposed representations, a ConvNet based
method is developed for action recognition. The image-based representations
enable us to fine-tune the existing Convolutional Neural Network (ConvNet)
models trained on image data without training a large number of parameters from
scratch. The proposed method achieved the state-of-art results on three large
datasets, namely, the Large-scale Continuous Gesture Recognition Dataset (means
Jaccard index 0.4109), the Large-scale Isolated Gesture Recognition Dataset
(59.21%), and the NTU RGB+D Dataset (87.08% cross-subject and 84.22%
cross-view) even though only the depth modality was used.

本论文提出了三种简单、紧凑而有效的深度序列表示，称为 DDI、DDNI 和 DDMNI，用于孤立和连续行为识别。在这些动态图像上，设计了基于 ConvNet 的方法进行行动识别，取得了三个大型数据集上的最先进结果，即 Large-scale Continuous Gesture Recognition Dataset（Jaccard 指数 0.4109）、Large-scale Isolated Gesture Recognition Dataset（59.21％）和 NTU RGB + D Dataset（跨主体 87.08％和跨视角 84.22％），即使只使用深度模态。

基于深度池化的卷积神经网络大规模三维动作识别

Depth Pooling Based Large-scale 3D Action Recognition with Convolutional  Neural Networks

Existing RNN-based approaches for action recognition from depth sequences
require either skeleton joints or hand-crafted depth features as inputs. An
end-to-end manner, mapping from raw depth maps to action classes, is
non-trivial to design due to the fact that: 1) single channel map lacks texture
thus weakens the discriminative power; 2) relatively small set of depth
training data. To address these challenges, we propose to learn an RNN driven
by privileged information (PI) in three-steps: An encoder is pre-trained to
learn a joint embedding of depth appearance and PI (i.e. skeleton joints). The
learned embedding layers are then tuned in the learning step, aiming to
optimize the network by exploiting PI in a form of multi-task loss. However,
exploiting PI as a secondary task provides little help to improve the
performance of a primary task (i.e. classification) due to the gap between
them. Finally, a bridging matrix is defined to connect two tasks by discovering
latent PI in the refining step. Our PI-based classification loss maintains a
consistency between latent PI and predicted distribution. The latent PI and
network are iteratively estimated and updated in an expectation-maximization
procedure. The proposed learning process provides greater discriminative power
to model subtle depth difference, while helping avoid overfitting the scarcer
training data. Our experiments show significant performance gains over
state-of-the-art methods on three public benchmark datasets and our newly
collected Blanket dataset.

本文提出了一种基于特权信息（PI）的深度序列动作识别方法，通过多任务学习将 PI 作为辅助任务引入，提高了模型的鉴别能力和训练效果。

基于特权信息的深度序列动作识别 RNN 的学习和优化

Learning and Refining of Privileged Information-based RNNs for Action  Recognition from Depth Sequences

This paper proposes three simple, compact yet effective representations of
depth sequences, referred to respectively as Dynamic Depth Images (DDI),
Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images
(DDMNI). These dynamic images are constructed from a sequence of depth maps
using bidirectional rank pooling to effectively capture the spatial-temporal
information. Such image-based representations enable us to fine-tune the
existing ConvNets models trained on image data for classification of depth
sequences, without introducing large parameters to learn. Upon the proposed
representations, a convolutional Neural networks (ConvNets) based method is
developed for gesture recognition and evaluated on the Large-scale Isolated
Gesture Recognition at the ChaLearn Looking at People (LAP) challenge 2016. The
method achieved 55.57\% classification accuracy and ranked $2^{nd}$ place in
this challenge but was very close to the best performance even though we only
used depth data.

该论文提出了三种简单、紧凑而又有效的深度序列表示方法，分别称为动态深度图像 (DDI)、动态深度法线图像 (DDNI) 和动态深度运动法线图像 (DDMNI)。这些动态图像是从一系列深度图像构建而成的，使用双向排名池化来有效地捕捉时空信息，从而使我们能够对已经训练好的卷积神经网络 (ConvNets) 模型进行微调，以用于深度序列的分类。在该论文提出的基础上，我们开发了一个卷积神经网络 (ConvNets) 方法，用于手势识别，并在 2016 年 ChaLearn Looking at People（LAP）挑战赛的大规模隔离手势识别中进行了评估，达到了 55.57％的分类准确率，并排名第二，虽然只使用了深度数据，但非常接近最佳表现。