Recent progresses on self-supervised 3D human action representation learning
are largely attributed to contrastive learning. However, in conventional
contrastive frameworks, the rich complementarity between different skeleton
modalities remains under-explored. Moreover, optimized with distinguishing
self-augmented samples, models struggle with numerous similar positive
instances in the case of limited action categories. In this work, we tackle the
aforementioned problems by introducing a general Inter- and Intra-modal Mutual
Distillation (I$^2$MD) framework. In I$^2$MD, we first re-formulate the
cross-modal interaction as a Cross-modal Mutual Distillation (CMD) process.
Different from existing distillation solutions that transfer the knowledge of a
pre-trained and fixed teacher to the student, in CMD, the knowledge is
continuously updated and bidirectionally distilled between modalities during
pre-training. To alleviate the interference of similar samples and exploit
their underlying contexts, we further design the Intra-modal Mutual
Distillation (IMD) strategy, In IMD, the Dynamic Neighbors Aggregation (DNA)
mechanism is first introduced, where an additional cluster-level discrimination
branch is instantiated in each modality. It adaptively aggregates
highly-correlated neighboring features, forming local cluster-level
contrasting. Mutual distillation is then performed between the two branches for
cross-level knowledge exchange. Extensive experiments on three datasets show
that our approach sets a series of new records.

介绍了一种新的互模态和内模态的相互蒸馏框架，通过改进跨模态交互和解决相似样本干扰问题，实现了自监督学习，在三个数据集上取得了新的记录。

I$^2$MD：带有内外模态相互蒸馏的三维动作表征学习

I$^2$MD: 3D Action Representation Learning with Inter- and Intra-modal  Mutual Distillation

In 3D action recognition, there exists rich complementary information between
skeleton modalities. Nevertheless, how to model and utilize this information
remains a challenging problem for self-supervised 3D action representation
learning. In this work, we formulate the cross-modal interaction as a
bidirectional knowledge distillation problem. Different from classic
distillation solutions that transfer the knowledge of a fixed and pre-trained
teacher to the student, in this work, the knowledge is continuously updated and
bidirectionally distilled between modalities. To this end, we propose a new
Cross-modal Mutual Distillation (CMD) framework with the following designs. On
the one hand, the neighboring similarity distribution is introduced to model
the knowledge learned in each modality, where the relational information is
naturally suitable for the contrastive frameworks. On the other hand,
asymmetrical configurations are used for teacher and student to stabilize the
distillation process and to transfer high-confidence information between
modalities. By derivation, we find that the cross-modal positive mining in
previous works can be regarded as a degenerated version of our CMD. We perform
extensive experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II datasets.
Our approach outperforms existing self-supervised methods and sets a series of
new records. The code is available at: this https URL

本文提出了一种新的跨模态交互知识蒸馏框架 Cross-modal Mutual Distillation (CMD) 来学习自监督的三维动作表示，该框架中引入邻域相似性分布来建模每种模态中学到的知识，并使用非对称配置来稳定蒸馏过程和在模态之间传输高置信度的信息，该方法在多个数据集上均举行了大量实验，超出了现有的自监督方法并创造了一系列新记录。