Current facial expression recognition (FER) models are often designed in a
supervised learning manner thus are constrained by the lack of large-scale
facial expression images with high-quality annotations. Consequently, these
models often fail to generalize well, performing poorly on unseen images in
training. Vision-language-based zero-shot models demonstrate a promising
potential for addressing such challenges. However, these models lack
task-specific knowledge therefore are not optimized for the nuances of
recognizing facial expressions. To bridge this gap, this work proposes a novel
method, Exp-CLIP, to enhance zero-shot FER by transferring the task knowledge
from large language models (LLMs). Specifically, based on the pre-trained
vision-language encoders, we incorporate a projection head designed to map the
initial joint vision-language space into a space that captures representations
of facial actions. To train this projection head for subsequent zero-shot
predictions, we propose to align the projected visual representations with
task-specific semantic meanings derived from the LLM encoder, and the text
instruction-based strategy is employed to customize the LLM knowledge. Given
unlabelled facial data and efficient training of the projection head, Exp-CLIP
achieves superior zero-shot results to the CLIP models and several other large
vision-language models (LVLMs) on seven in-the-wild FER datasets. The code and
pre-trained models are available at
https://github.com/zengqunzhao/Exp-CLIP.

本研究提出了一种名为 Exp-CLIP 的新方法，通过从大型语言模型（LLMs）中转移任务知识来增强零样例人脸表情识别。利用预训练的视觉 - 语言编码器，通过投影头将初始联合视觉 - 语言空间映射到捕捉面部动作表示的空间，以此训练投影头进行零样例预测，同时采用基于文本指令的策略定制 LLM 知识。Exp-CLIP 在七个野外人脸表情数据集上实现了优于 CLIP 模型和其他若干大型视觉 - 语言模型（LVLMs）的零样例结果。