We present Affect2MM, a learning method for time-series emotion prediction
for multimedia content. Our goal is to automatically capture the varying
emotions depicted by characters in real-life human-centric situations and
behaviors. We use the ideas from emotion causation theories to computationally
model and determine the emotional state evoked in clips of movies. Affect2MM
explicitly models the temporal causality using attention-based methods and
Granger causality. We use a variety of components like facial features of
actors involved, scene understanding, visual aesthetics, action/situation
description, and movie script to obtain an affective-rich representation to
understand and perceive the scene. We use an LSTM-based learning model for
emotion perception. To evaluate our method, we analyze and compare our
performance on three datasets, SENDv1, MovieGraphs, and the LIRIS-ACCEDE
dataset, and observe an average of 10-15% increase in the performance over SOTA
methods for all three datasets.

通过 Affect2MM，我们提供了一种用于多媒体内容时间序列情感预测的学习方法，利用情感因果理论来计算实现对电影片段中情感的显式建模，通过基于注意力机制和格兰杰因果性的方法来显式建模情感的时间因果性，并使用 LSTM 的情感感知模型进行评估。