Unsupervised pre-training has shown great success in skeleton-based action
understanding recently. Existing works typically train separate
modality-specific models, then integrate the multi-modal information for action
understanding by a late-fusion strategy. Although these approaches have
achieved significant performance, they suffer from the complex yet redundant
multi-stream model designs, each of which is also limited to the fixed input
skeleton modality. To alleviate these issues, in this paper, we propose a
Unified Multimodal Unsupervised Representation Learning framework, called
UmURL, which exploits an efficient early-fusion strategy to jointly encode the
multi-modal features in a single-stream manner. Specifically, instead of
designing separate modality-specific optimization processes for uni-modal
unsupervised learning, we feed different modality inputs into the same stream
with an early-fusion strategy to learn their multi-modal features for reducing
model complexity. To ensure that the fused multi-modal features do not exhibit
modality bias, i.e., being dominated by a certain modality input, we further
propose both intra- and inter-modal consistency learning to guarantee that the
multi-modal features contain the complete semantics of each modal via feature
decomposition and distinct alignment. In this manner, our framework is able to
learn the unified representations of uni-modal or multi-modal skeleton input,
which is flexible to different kinds of modality input for robust action
understanding in practical cases. Extensive experiments conducted on three
large-scale datasets, i.e., NTU-60, NTU-120, and PKU-MMD II, demonstrate that
UmURL is highly efficient, possessing the approximate complexity with the
uni-modal methods, while achieving new state-of-the-art performance across
various downstream task scenarios in skeleton-based action representation
learning.

无监督预训练及早融合策略的统一多模态无监督表示学习框架（UmURL）旨在提取骨架输入的统一表示，通过特征分解和对齐保证多模态特征包含各模态的完整语义，并在各种场景中实现鲁棒动作理解。该框架在三个大规模数据集上取得了新的多种下游任务场景中骨架 - based 动作表示学习的最先进性能。