Facial Expression Recognition (FER) is a crucial task in affective computing,
but its conventional focus on the seven basic emotions limits its applicability
to the complex and expanding emotional spectrum. To address the issue of new
and unseen emotions present in dynamic in-the-wild FER, we propose a novel
vision-language model that utilises sample-level text descriptions (i.e.
captions of the context, expressions or emotional cues) as natural language
supervision, aiming to enhance the learning of rich latent representations, for
zero-shot classification. To test this, we evaluate using zero-shot
classification of the model trained on sample-level descriptions on four
popular dynamic FER datasets. Our findings show that this approach yields
significant improvements when compared to baseline methods. Specifically, for
zero-shot video FER, we outperform CLIP by over 10\% in terms of Weighted
Average Recall and 5\% in terms of Unweighted Average Recall on several
datasets. Furthermore, we evaluate the representations obtained from the
network trained using sample-level descriptions on the downstream task of
mental health symptom estimation, achieving performance comparable or superior
to state-of-the-art methods and strong agreement with human experts. Namely, we
achieve a Pearson's Correlation Coefficient of up to 0.85 on schizophrenia
symptom severity estimation, which is comparable to human experts' agreement.
The code is publicly available at: this https URL

通过采用样本级文本描述（即上下文、表情或情感线索的标题）作为自然语言监督，我们提出了一种新颖的视觉 - 语言模型，旨在增强丰富的潜在表示的学习，以进行零样本分类。通过对四个流行的动态情感识别数据集进行模型测试，我们发现与基线方法相比，该方法在零样本视频情感识别方面表现出显著的改进，并在精神健康症状估计等下游任务中取得与人类专家相当或更优的性能。