Human-centric perception plays a vital role in vision and graphics. But their
data annotations are prohibitively expensive. Therefore, it is desirable to
have a versatile pre-train model that serves as a foundation for data-efficient
downstream tasks transfer. To this end, we propose the Human-Centric
Multi-Modal Contrastive Learning framework HCMoCo that leverages the
multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective
representation learning. The objective comes with two main challenges: dense
pre-train for multi-modality data, efficient usage of sparse human priors. To
tackle the challenges, we design the novel Dense Intra-sample Contrastive
Learning and Sparse Structure-aware Contrastive Learning targets by
hierarchically learning a modal-invariant latent space featured with continuous
and ordinal feature distribution and structure-aware semantic consistency.
HCMoCo provides pre-train for different modalities by combining heterogeneous
datasets, which allows efficient usage of existing task-specific human data.
Extensive experiments on four downstream tasks of different modalities
demonstrate the effectiveness of HCMoCo, especially under data-efficient
settings (7.16% and 12% improvement on DensePose Estimation and Human Parsing).
Moreover, we demonstrate the versatility of HCMoCo by exploring cross-modality
supervision and missing-modality inference, validating its strong ability in
cross-modal association and reasoning.

本文提出 Human-Centric Multi-Modal Contrastive Learning 框架，通过密集内部样本对比学习和稀疏结构感知对比学习目标，实现多视角数据有效特征表示，为数据高效下游任务转移提供了通用的预训练模型。