In Video-based Facial Expression Recognition (V-FER), models are typically
trained on closed-set datasets with a fixed number of known classes. However,
these V-FER models cannot deal with unknown classes that are prevalent in
real-world scenarios. In this paper, we introduce a challenging Open-set
Video-based Facial Expression Recognition (OV-FER) task, aiming at identifying
not only known classes but also new, unknown human facial expressions not
encountered during training. While existing approaches address open-set
recognition by leveraging large-scale vision-language models like CLIP to
identify unseen classes, we argue that these methods may not adequately capture
the nuanced and subtle human expression patterns required by the OV-FER task.
To address this limitation, we propose a novel Human Expression-Sensitive
Prompting (HESP) mechanism to significantly enhance CLIP's ability to model
video-based facial expression details effectively, thereby presenting a new
CLIP-based OV-FER approach. Our proposed HESP comprises three components: 1) a
textual prompting module with learnable prompt representations to complement
the original CLIP textual prompts and enhance the textual representations of
both known and unknown emotions, 2) a visual prompting module that encodes
temporal emotional information from video frames using expression-sensitive
attention, equipping CLIP with a new visual modeling ability to extract
emotion-rich information, 3) a delicately designed open-set multi-task learning
scheme that facilitates prompt learning and encourages interactions between the
textual and visual prompting modules. Extensive experiments conducted on four
OV-FER task settings demonstrate that HESP can significantly boost CLIP's
performance (a relative improvement of 17.93% on AUROC and 106.18% on OSCR) and
outperform other state-of-the-art open-set video understanding methods by a
large margin.

在本文中，我们提出了一种针对开放场景的视频人脸表情识别 (OV-FER) 任务，旨在识别不仅包含已知类别，也包含训练中未遇到的新的未知人脸表情。为了克服现有方法对 OV-FER 任务所需的微妙和细微人脸表情模式的不足，我们提出了一种新颖的人脸表情敏感引导 (HESP) 机制来显著增强 CLIP 模型对视频人脸表情细节的建模能力。在四个 OV-FER 任务设置上进行的广泛实验表明，HESP 可以显著提高 CLIP 的性能（在 AUROC 上相对提升了 17.93%，在 OSCR 上相对提升了 106.18%），并大幅优于其他最先进的开放式视频理解方法。