Skeleton sequence representation learning has shown great advantages for action recognition due to its promising ability to model human joints and topology. However, the current methods usually require sufficient labeled data for training computationally expensive models, which is labor-intensive and time-consuming. Moreover, these methods ignore how to utilize the fine-grained dependencies among different skeleton joints to pre-train an efficient skeleton sequence learning model that can generalize well across different datasets. In this paper, we propose an efficient skeleton sequence learning framework, named Skeleton Sequence Learning (SSL). To comprehensively capture the human pose and obtain discriminative skeleton sequence representation, we build an asymmetric graph-based encoder-decoder pre-training architecture named SkeletonMAE, which embeds skeleton joint sequence into Graph Convolutional Network (GCN) and reconstructs the masked skeleton joints and edges based on the prior human topology knowledge. Then, the pre-trained SkeletonMAE encoder is integrated with the Spatial-Temporal Representation Learning (STRL) module to build the SSL framework. Extensive experimental results show that our SSL generalizes well across different datasets and outperforms the state-of-the-art self-supervised skeleton-based action recognition methods on FineGym, Diving48, NTU 60 and NTU 120 datasets. Additionally, we obtain comparable performance to some fully supervised methods. The code is avaliable at https://github.com/HongYan1123/SkeletonMAE.

本文提出了一种高效的骨架序列学习框架，名为Skeleton Sequence Learning（SSL）。通过将骨架关节序列嵌入图卷积网络（GCN）并基于先验人体拓扑知识重构遮蔽骨架关节和边，我们构建了一个不对称的基于图的编码器-解码器预训练架构，命名为SkeletonMAE，然后将预训练的SkeletonMAE编码器与时空表示学习（STRL）模块集成到SSL框架中。广泛的实验结果表明，我们的SSL在不同数据集上具有良好的泛化能力，在FineGym、Diving48、NTU 60和NTU 120数据集上优于最先进的自监督骨骼动作识别方法，同时与某些完全监督方法具有可比较的性能。

SkeletonMAE：基于图的骨架序列预训练掩蔽自编码器