Recognising emotions in context involves identifying the apparent emotions of
an individual, taking into account contextual cues from the surrounding scene.
Previous approaches to this task have involved the design of explicit
scene-encoding architectures or the incorporation of external scene-related
information, such as captions. However, these methods often utilise limited
contextual information or rely on intricate training pipelines. In this work,
we leverage the groundbreaking capabilities of Vision-and-Large-Language Models
(VLLMs) to enhance in-context emotion classification without introducing
complexity to the training process in a two-stage approach. In the first stage,
we propose prompting VLLMs to generate descriptions in natural language of the
subject's apparent emotion relative to the visual context. In the second stage,
the descriptions are used as contextual information and, along with the image
input, are used to train a transformer-based architecture that fuses text and
visual features before the final classification task. Our experimental results
show that the text and image features have complementary information, and our
fused architecture significantly outperforms the individual modalities without
any complex training methods. We evaluate our approach on three different
datasets, namely, EMOTIC, CAER-S, and BoLD, and achieve state-of-the-art or
comparable accuracy across all datasets and metrics compared to much more
complex approaches. The code will be made publicly available on github:
this https URL

利用视觉和大规模语言模型（VLLMs）的开创性能力，我们提出了一种两阶段方法来增强上下文情绪分类，首先利用 VLLMs 生成自然语言描述主体与视觉情境之间的明显情绪，然后将这些描述与图像输入一起用于训练融合文本和视觉特征的基于变换器的结构，在不引入复杂训练方法的情况下显著优于单一模态，并且在 EMOTIC、CAER-S 和 BoLD 三个数据集上达到了与更复杂方法相媲美甚至更高的准确性。