Self-supervised representation learning for human action recognition has
developed rapidly in recent years. Most of the existing works are based on
skeleton data while using a multi-modality setup. These works overlooked the
differences in performance among modalities, which led to the propagation of
erroneous knowledge between modalities while only three fundamental modalities,
i.e., joints, bones, and motions are used, hence no additional modalities are
explored.
In this work, we first propose an Implicit Knowledge Exchange Module (IKEM)
which alleviates the propagation of erroneous knowledge between low-performance
modalities. Then, we further propose three new modalities to enrich the
complementary information between modalities. Finally, to maintain efficiency
when introducing new modalities, we propose a novel teacher-student framework
to distill the knowledge from the secondary modalities into the mandatory
modalities considering the relationship constrained by anchors, positives, and
negatives, named relational cross-modality knowledge distillation. The
experimental results demonstrate the effectiveness of our approach, unlocking
the efficient use of skeleton-based multi-modality data. Source code will be
made publicly available at this https URL

自我监督表示学习在人体动作识别方面得到了快速发展。本文首先提出了一种隐式知识交换模块（IKEM），用于减轻低性能模态之间的错误知识传播。然后，我们进一步提出了三种新模态来丰富不同模态之间的互补信息。最后，通过锚点、正样本和负样本之间的约束关系，我们提出了一种新的教师 - 学生框架，将次要模态的知识蒸馏到强制性模态中，名为关系型跨模态知识蒸馏。实验证明了我们方法的有效性，可以高效利用基于骨架的多模态数据。源代码将公开发布在给定的网址。