Emotion estimation in images is a challenging task, typically using computer
vision methods to directly estimate people's emotions using face, body pose and
contextual cues. In this paper, we explore whether Large Language Models (LLMs)
can support the contextual emotion estimation task, by first captioning images,
then using an LLM for inference. First, we must understand: how well do LLMs
perceive human emotions? And which parts of the information enable them to
determine emotions? One initial challenge is to construct a caption that
describes a person within a scene with information relevant for emotion
perception. Towards this goal, we propose a set of natural language descriptors
for faces, bodies, interactions, and environments. We use them to manually
generate captions and emotion annotations for a subset of 331 images from the
EMOTIC dataset. These captions offer an interpretable representation for
emotion estimation, towards understanding how elements of a scene affect
emotion perception in LLMs and beyond. Secondly, we test the capability of a
large language model to infer an emotion from the resulting image captions. We
find that GPT-3.5, specifically the text-davinci-003 model, provides
surprisingly reasonable emotion predictions consistent with human annotations,
but accuracy can depend on the emotion concept. Overall, the results suggest
promise in the image captioning and LLM approach.

通过图像的标题生成以及使用大型语言模型（LLMs）进行推理的方法，该论文探讨了 LLMs 是否可以支持上下文情感估计任务，并表明 GPT-3.5 模型在情感预测方面具有潜力。

图像标题的情感背景评估

Contextual Emotion Estimation from Image Captions

The continuous improvement of human-computer interaction technology makes it
possible to compute emotions. In this paper, we introduce our submission to the
CVPR 2023 Competition on Affective Behavior Analysis in-the-wild (ABAW).
Sentiment analysis in human-computer interaction should, as far as possible
Start with multiple dimensions, fill in the single imperfect emotion channel,
and finally determine the emotion tendency by fitting multiple results.
Therefore, We exploited multimodal features extracted from video of different
lengths from the competition dataset, including audio, pose and images.
Well-informed emotion representations drive us to propose a Attention-based
multimodal framework for emotion estimation. Our system achieves the
performance of 0.361 on the validation dataset. The code is available at
[this https URL].

本文介绍了一个基于多模态的情感估计框架，通过提取比赛数据集中的不同长度的视频的多模态特征，包括音频、姿态和图像，确定情感趋势。该系统在验证数据集中达到 0.361 的性能。

视频中多模态特征提取和基于注意力的融合情绪估计

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Among the different modalities to assess emotion, electroencephalogram (EEG),
representing the electrical brain activity, achieved motivating results over
the last decade. Emotion estimation from EEG could help in the diagnosis or
rehabilitation of certain diseases. In this paper, we propose a dual model
considering two different representations of EEG feature maps: 1) a sequential
based representation of EEG band power, 2) an image-based representation of the
feature vectors. We also propose an innovative method to combine the
information based on a saliency analysis of the image-based model to promote
joint learning of both model parts. The model has been evaluated on four
publicly available datasets: SEED-IV, SEED, DEAP and MPED. The achieved results
outperform results from state-of-the-art approaches for three of the proposed
datasets with a lower standard deviation that reflects higher stability. For
sake of reproducibility, the codes and models proposed in this paper are
available at this https URL.

本文提出了一种双模型方法，结合了基于序列的 EEG 幅值表示和基于图像的表示。采用基于显著性分析的图像模型，促进了两个模型部分的联合学习。在四个公开数据集上进行评估，结果表明本方法优于现有方法，在情感估计任务上取得了较高的稳定性。

一种基于显著性特征融合的 EEG 情感估计模型

A Saliency based Feature Fusion Model for EEG Emotion Estimation

Emotion estimation in music listening is confronting challenges to capture
the emotion variation of listeners. Recent years have witnessed attempts to
exploit multimodality fusing information from musical contents and
physiological signals captured from listeners to improve the performance of
emotion recognition. In this paper, we present a study of fusion of signals of
electroencephalogram (EEG), a tool to capture brainwaves at a high-temporal
resolution, and musical features at decision level in recognizing the
time-varying binary classes of arousal and valence. Our empirical results
showed that the fusion could outperform the performance of emotion recognition
using only EEG modality that was suffered from inter-subject variability, and
this suggested the promise of multimodal fusion in improving the accuracy of
music-emotion recognition.

本文研究了如何在音乐欣赏中通过脑电图和音乐特征相结合的多模态融合方法提高情感识别的准确性，结果表明多模态融合方法能够优于单一模态，并提出了未来融合研究的方向。