"How does the person in the bounding box feel?" Achieving human-level
recognition of the apparent emotion of a person in real world situations
remains an unsolved task in computer vision. Facial expressions are not enough:
body pose, contextual knowledge, and commonsense reasoning all contribute to
how humans perform this emotional theory of mind task. In this paper, we
examine two major approaches enabled by recent large vision language models: 1)
image captioning followed by a language-only LLM, and 2) vision language
models, under zero-shot and fine-tuned setups. We evaluate the methods on the
Emotions in Context (EMOTIC) dataset and demonstrate that a vision language
model, fine-tuned even on a small dataset, can significantly outperform
traditional baselines. The results of this work aim to help robots and agents
perform emotionally sensitive decision-making and interaction in the future.

在这篇研究论文中，我们使用最近的大型视觉语言模型来探讨两种主要方法：图像字幕生成与仅使用语言的 LLM，以及零样本和微调设置下的视觉语言模型。我们在 Emotions in Context（EMOTIC）数据集上评估这些方法，并展示出即使在小型数据集上进行微调，视觉语言模型的性能也能显著超过传统的基准方法。我们的研究结果旨在帮助未来的机器人和智能系统对情感进行敏感的决策和交互行为。