Many attempts have been made towards combining RGB and 3D poses for the
recognition of Activities of Daily Living (ADL). ADL may look very similar and
often necessitate to model fine-grained details to distinguish them. Because
the recent 3D ConvNets are too rigid to capture the subtle visual patterns
across an action, this research direction is dominated by methods combining RGB
and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the
absence of appropriate sensors. This limits the usage of aforementioned
approaches in real-world applications requiring low latency. Then, how to best
take advantage of 3D Poses for recognizing ADL? To this end, we propose an
extension of a pose driven attention mechanism: Video-Pose Network (VPN),
exploring two distinct directions. One is to transfer the Pose knowledge into
RGB through a feature-level distillation and the other towards mimicking pose
driven attention through an attention-level distillation. Finally, these two
approaches are integrated into a single model, we call VPN++. We show that
VPN++ is not only effective but also provides a high speed up and high
resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the
representative baselines on 4 public datasets. Code is available at
this https URL

提出了 Video-Pose Network (VPN) 用于识别日常活动，该模型包含两个不同的方向：一个是通过特征层面的蒸馏将 Pose 知识转移到 RGB 中，另一个是通过注意层面的蒸馏模仿姿势驱动的注意。该模型在 4 个公共数据集上表现出了高速且具有噪声姿势的鲁棒性。

VPN++：重新思考视频姿态嵌入以理解日常活动

VPN++: Rethinking Video-Pose embeddings for understanding Activities of  Daily Living

In this paper, we focus on the spatio-temporal aspect of recognizing
Activities of Daily Living (ADL). ADL have two specific properties (i) subtle
spatio-temporal patterns and (ii) similar visual patterns varying with time.
Therefore, ADL may look very similar and often necessitate to look at their
fine-grained details to distinguish them. Because the recent spatio-temporal 3D
ConvNets are too rigid to capture the subtle visual patterns across an action,
we propose a novel Video-Pose Network: VPN. The 2 key components of this VPN
are a spatial embedding and an attention network. The spatial embedding
projects the 3D poses and RGB cues in a common semantic space. This enables the
action recognition framework to learn better spatio-temporal features
exploiting both modalities. In order to discriminate similar actions, the
attention network provides two functionalities - (i) an end-to-end learnable
pose backbone exploiting the topology of human body, and (ii) a coupler to
provide joint spatio-temporal attention weights across a video. Experiments
show that VPN outperforms the state-of-the-art results for action
classification on a large scale human activity dataset: NTU-RGB+D 120, its
subset NTU-RGB+D 60, a real-world challenging human activity dataset: Toyota
Smarthome and a small scale human-object interaction dataset Northwestern UCLA.

本文提出一种新的 Video-Pose Network（VPN）框架，其中空间嵌入和注意力网络为其两个关键组成部分。在多个数据集上的实验证明，VPN 优于现有技术在动作识别方面的性能，并能更好地利用 ADL 的细微视觉模式来提高识别能力。