In 3D human action recognition, limited supervised data makes it challenging
to fully tap into the modeling potential of powerful networks such as
transformers. As a result, researchers have been actively investigating
effective self-supervised pre-training strategies. In this work, we show that
instead of following the prevalent pretext task to perform masked
self-component reconstruction in human joints, explicit contextual motion
modeling is key to the success of learning effective feature representation for
3D action recognition. Formally, we propose the Masked Motion Prediction (MAMP)
framework. To be specific, the proposed MAMP takes as input the masked
spatio-temporal skeleton sequence and predicts the corresponding temporal
motion of the masked human joints. Considering the high temporal redundancy of
the skeleton sequence, in our MAMP, the motion information also acts as an
empirical semantic richness prior that guide the masking process, promoting
better attention to semantically rich temporal regions. Extensive experiments
on NTU-60, NTU-120, and PKU-MMD datasets show that the proposed MAMP
pre-training substantially improves the performance of the adopted vanilla
transformer, achieving state-of-the-art results without bells and whistles. The
source code of our MAMP is available at this https URL

本文提出了一种用于 3D 动作识别的 Masked Motion Prediction（MAMP）框架，通过对蒙面的空间 - 时间骨骼序列进行预测，实现对丰富语义区域的更好关注，进而提高自监督预训练的性能。经过对 NTU-60、NTU-120 和 PKU-MMD 数据集的广泛实验，表明所提出的 MAMP 预训练方法显著提高了基础的 transformer 模型的性能，并取得了最先进的结果。MAMP 的源代码可以在给定的网址上找到。

面具化运动预测器是强大的三维动作表示学习器

Masked Motion Predictors are Strong 3D Action Representation Learners

Action recognition has been a heated topic in computer vision for its wide
application in vision systems. Previous approaches achieve improvement by
fusing the modalities of the skeleton sequence and RGB video. However, such
methods have a dilemma between the accuracy and efficiency for the high
complexity of the RGB video network. To solve the problem, we propose a
multi-modality feature fusion network to combine the modalities of the skeleton
sequence and RGB frame instead of the RGB video, as the key information
contained by the combination of skeleton sequence and RGB frame is close to
that of the skeleton sequence and RGB video. In this way, the complementary
information is retained while the complexity is reduced by a large margin. To
better explore the correspondence of the two modalities, a two-stage fusion
framework is introduced in the network. In the early fusion stage, we introduce
a skeleton attention module that projects the skeleton sequence on the single
RGB frame to help the RGB frame focus on the limb movement regions. In the late
fusion stage, we propose a cross-attention module to fuse the skeleton feature
and the RGB feature by exploiting the correlation. Experiments on two
benchmarks NTU RGB+D and SYSU show that the proposed model achieves competitive
performance compared with the state-of-the-art methods while reduces the
complexity of the network.

本文研究动作识别，提出一种基于多模态特征融合网络的方法，将骨架序列和 RGB 帧的关键信息相结合，以此保留补充信息的同时大大降低了其网络的复杂性。实验表明，所提出的模型能够与现有最先进的方法相比取得竞争性的表现。