Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in
recent years for its critical role in creating emotion-ware intelligent
machines. Previous efforts in this area are dominated by the supervised
learning paradigm. Despite significant progress, supervised learning is meeting
its bottleneck due to the longstanding data scarcity issue in AVER. Motivated
by recent advances in self-supervised learning, we propose Hierarchical
Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that
leverages large-scale self-supervised pre-training on vast unlabeled
audio-visual data to promote the advancement of AVER. Following prior arts in
self-supervised audio-visual representation learning, HiCMAE adopts two primary
forms of self-supervision for pre-training, namely masked data modeling and
contrastive learning. Unlike them which focus exclusively on top-layer
representations while neglecting explicit guidance of intermediate layers,
HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual
feature learning and improve the overall quality of learned representations. To
verify the effectiveness of HiCMAE, we conduct extensive experiments on 9
datasets covering both categorical and dimensional AVER tasks. Experimental
results show that our method significantly outperforms state-of-the-art
supervised and self-supervised audio-visual methods, which indicates that
HiCMAE is a powerful audio-visual emotion representation learner. Codes and
models will be publicly available at this https URL

提出了一种基于自监督学习的 Hierarchical Contrastive Masked Autoencoder（HiCMAE）框架，通过大规模的自监督预训练提升了音视频情感识别的性能，实验证明 HiCMAE 方法在多个数据集上显著优于现有的有监督和自监督音视频方法。