This paper presents a novel visual-language model called DFER-CLIP, which is
based on the CLIP model and designed for in-the-wild Dynamic Facial Expression
Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual
part and a textual part. For the visual part, based on the CLIP image encoder,
a temporal model consisting of several Transformer encoders is introduced for
extracting temporal facial expression features, and the final feature embedding
is obtained as a learnable "class" token. For the textual part, we use as
inputs textual descriptions of the facial behaviour that is related to the
classes (facial expressions) that we are interested in recognising -- those
descriptions are generated using large language models, like ChatGPT. This, in
contrast to works that use only the class names and more accurately captures
the relationship between them. Alongside the textual description, we introduce
a learnable token which helps the model learn relevant context information for
each expression during training. Extensive experiments demonstrate the
effectiveness of the proposed method and show that our DFER-CLIP also achieves
state-of-the-art results compared with the current supervised DFER methods on
the DFEW, FERV39k, and MAFW benchmarks. Code is publicly available at
this https URL

该论文介绍了一种名为 DFER-CLIP 的新型视觉语言模型，它基于 CLIP 模型，用于野外动态面部表情识别 (DFER)。该模型包括视觉部分和文本部分，通过提取时序面部表情特征和学习上下文信息，实现了与当前监督 DFER 方法相比的最先进结果。