The emotional theory of mind problem in images is an emotion recognition
task, specifically asking "How does the person in the bounding box feel?"
Facial expressions, body pose, contextual information and implicit commonsense
knowledge all contribute to the difficulty of the task, making this task
currently one of the hardest problems in affective computing. The goal of this
work is to evaluate the emotional commonsense knowledge embedded in recent
large vision language models (CLIP, LLaVA) and large language models (GPT-3.5)
on the Emotions in Context (EMOTIC) dataset. In order to evaluate a purely
text-based language model on images, we construct "narrative captions" relevant
to emotion perception, using a set of 872 physical social signal descriptions
related to 26 emotional categories, along with 224 labels for emotionally
salient environmental contexts, sourced from writer's guides for character
expressions and settings. We evaluate the use of the resulting captions in an
image-to-language-to-emotion task. Experiments using zero-shot vision-language
models on EMOTIC show that combining "fast" and "slow" reasoning is a promising
way forward to improve emotion recognition systems. Nevertheless, a gap remains
in the zero-shot emotional theory of mind task compared to prior work trained
on the EMOTIC dataset.

在 Emotions in Context（EMOTIC）数据集上，评估了最近的大规模视觉语言模型（CLIP，LLaVA）和大语言模型（GPT-3.5）中嵌入的情绪常识知识。使用一组 872 个与 26 个情绪类别相关的物理社交信号描述和 224 个情绪显著环境背景描述构建了与情绪感知相关的 “叙述性字幕”，并评估了在图像 - 语言 - 情感任务中使用这些字幕的效果。零样本视觉语言模型的实验表明，结合 “快速” 和 “慢速” 推理是改进情绪识别系统的一种有前途的方法，但与在 EMOTIC 数据集上训练的以往研究相比，零样本情绪心理理论任务仍存在差距。