Cross-modality distillation arises as an important topic for data modalities
containing limited knowledge such as depth maps and high-quality sketches. Such
techniques are of great importance, especially for memory and
privacy-restricted scenarios where labeled training data is generally
unavailable. To solve the problem, existing label-free methods leverage a few
pairwise unlabeled data to distill the knowledge by aligning features or
statistics between the source and target modalities. For instance, one
typically aims to minimize the L2 distance or contrastive loss between the
learned features of pairs of samples in the source (e.g. image) and the target
(e.g. sketch) modalities. However, most algorithms in this domain only focus on
the experimental results but lack theoretical insight. To bridge the gap
between the theory and practical method of cross-modality distillation, we
first formulate a general framework of cross-modality contrastive distillation
(CMCD), built upon contrastive learning that leverages both positive and
negative correspondence, towards a better distillation of generalizable
features. Furthermore, we establish a thorough convergence analysis that
reveals that the distance between source and target modalities significantly
impacts the test error on downstream tasks within the target modality which is
also validated by the empirical results. Extensive experimental results show
that our algorithm outperforms existing algorithms consistently by a margin of
2-3\% across diverse modalities and tasks, covering modalities of image,
sketch, depth map, and audio and tasks of recognition and segmentation.

跨模态蒸馏是一个重要主题，用于包含有限知识的数据模态，如深度图和高质量素描。我们提出了一个基于对比学习的跨模态对比蒸馏（CMCD）的通用框架，该框架利用正负对应关系更好地蒸馏可泛化的特征，并在实验结果和收敛分析中验证了源模态和目标模态之间的距离对目标模态下游任务的测试误差有着显著影响。

跨模态对比学习蒸馏理论研究

On the Theory of Cross-Modality Distillation with Contrastive Learning

The success of supervised deep learning models on cell recognition tasks
relies on detailed annotations. Many previous works have managed to reduce the
dependency on labels. However, considering the large number of cells contained
in a patch, costly and inefficient labeling is still inevitable. To this end,
we explored label-free methods for cell recognition. Prior self-activation maps
(PSM) are proposed to generate pseudo masks as training targets. To be
specific, an activation network is trained with self-supervised learning. The
gradient information in the shallow layers of the network is aggregated to
generate prior self-activation maps. Afterward, a semantic clustering module is
then introduced as a pipeline to transform PSMs to pixel-level semantic pseudo
masks for downstream tasks. We evaluated our method on two histological
datasets: MoNuSeg (cell segmentation) and BCData (multi-class cell detection).
Compared with other fully-supervised and weakly-supervised methods, our method
can achieve competitive performance without any manual annotations. Our simple
but effective framework can also achieve multi-class cell detection which can
not be done by existing unsupervised methods. The results show the potential of
PSMs that might inspire other research to deal with the hunger for labels in
medical area.

该论文的主要研究方向是关于使用无标签方法进行细胞识别，通过先前的自激活图生成伪掩膜作为训练目标，并引入语义聚类模块将先前的自激活图转换为像素级语义伪掩膜，在细胞分割和多类别细胞检测的数据集上取得了竞争性的性能，同时还展示了先前自激活图在解决医学领域标签不足问题方面的潜力。