The goal of this study is to develop and analyze multimodal models for
predicting experienced affective responses of viewers watching movie clips. We
develop hybrid multimodal prediction models based on both the video and audio
of the clips. For the video content, we hypothesize that both image content and
motion are crucial features for evoked emotion prediction. To capture such
information, we extract features from RGB frames and optical flow using
pre-trained neural networks. For the audio model, we compute an enhanced set of
low-level descriptors including intensity, loudness, cepstrum, linear predictor
coefficients, pitch and voice quality. Both visual and audio features are then
concatenated to create audio-visual features, which are used to predict the
evoked emotion. To classify the movie clips into the corresponding affective
response categories, we propose two approaches based on deep neural network
models. The first one is based on fully connected layers without memory on the
time component, the second incorporates the sequential dependency with a long
short-term memory recurrent neural network (LSTM). We perform a thorough
analysis of the importance of each feature set. Our experiments reveal that in
our set-up, predicting emotions at each time step independently gives slightly
better accuracy performance than with the LSTM. Interestingly, we also observe
that the optical flow is more informative than the RGB in videos, and overall,
models using audio features are more accurate than those based on video
features when making the final prediction of evoked emotions.

本研究旨在开发和分析多模态模型，预测观众观看电影片段时的情感反应。研究发现，通过提取 RGB 帧和光流的特征以及音频中的各种低级描述符，采用深度神经网络模型来进行情感预测更为精确。其中，光流特征比 RGB 视频更加信息含量丰富，且音频特征的预测精度高于视频特征。