Video emotion recognition is an important branch of affective computing, and
its solutions can be applied in different fields such as human-computer
interaction (HCI) and intelligent medical treatment. Although the number of
papers published in the field of emotion recognition is increasing, there are
few comprehensive literature reviews covering related research on video emotion
recognition. Therefore, this paper selects articles published from 2015 to 2023
to systematize the existing trends in video emotion recognition in related
studies. In this paper, we first talk about two typical emotion models, then we
talk about databases that are frequently utilized for video emotion
recognition, including unimodal databases and multimodal databases. Next, we
look at and classify the specific structure and performance of modern unimodal
and multimodal video emotion recognition methods, talk about the benefits and
drawbacks of each, and then we compare them in detail in the tables. Further,
we sum up the primary difficulties right now looked by video emotion
recognition undertakings and point out probably the most encouraging future
headings, such as establishing an open benchmark database and better multimodal
fusion strategys. The essential objective of this paper is to assist scholarly
and modern scientists with keeping up to date with the most recent advances and
new improvements in this speedy, high-influence field of video emotion
recognition.

本文选择 2015 年至 2023 年间发表的论文，系统总结了关于视频情感识别的现有研究趋势，讨论了典型情感模型、常用数据库以及现代单模态和多模态视频情感识别方法的结构和性能，并比较了它们的优缺点。此外，总结了目前视频情感识别项目所面临的主要困难，并指出了未来发展方向，如建立开放基准数据库和改进多模态融合策略。本文旨在帮助学术界和工业界的科学家及时了解视频情感识别领域的最新进展和新发展。

视频情绪识别综述

Emotion Recognition by Video: A review

Emotion recognition in user-generated videos plays an important role in
human-centered computing. Existing methods mainly employ traditional two-stage
shallow pipeline, i.e. extracting visual and/or audio features and training
classifiers. In this paper, we propose to recognize video emotions in an
end-to-end manner based on convolutional neural networks (CNNs). Specifically,
we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture
that integrates spatial, channel-wise, and temporal attentions into a visual 3D
CNN and temporal attentions into an audio 2D CNN. Further, we design a special
classification loss, i.e. polarity-consistent cross-entropy loss, based on the
polarity-emotion hierarchy constraint to guide the attention generation.
Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6
datasets demonstrate that the proposed VAANet outperforms the state-of-the-art
approaches for video emotion recognition. Our source code is released at:
this https URL

本研究提出了一种基于卷积神经网络的深层视听关注网络（VAANet）来进行用户生成视频中的情感识别，其结果在两个不易的视频情绪识别数据集上均优于现有的方法。

用户生成视频情感识别的端到端视听注意力网络

An End-to-End Visual-Audio Attention Network for Emotion Recognition in  User-Generated Videos

Spatial-temporal feature learning is of vital importance for video emotion
recognition. Previous deep network structures often focused on macro-motion
which extends over long time scales, e.g., on the order of seconds. We believe
integrating structures capturing information about both micro- and macro-motion
will benefit emotion prediction, because human perceive both micro- and
macro-expressions. In this paper, we propose to combine micro- and macro-motion
features to improve video emotion recognition with a two-stream recurrent
network, named MIMAMO (Micro-Macro-Motion) Net. Specifically, smaller and
shorter micro-motions are analyzed by a two-stream network, while larger and
more sustained macro-motions can be well captured by a subsequent recurrent
network. Assigning specific interpretations to the roles of different parts of
the network enables us to make choice of parameters based on prior knowledge:
choices that turn out to be optimal. One of the important innovations in our
model is the use of interframe phase differences rather than optical flow as
input to the temporal stream. Compared with the optical flow, phase differences
require less computation and are more robust to illumination changes. Our
proposed network achieves state of the art performance on two video emotion
datasets, the OMG emotion dataset and the Aff-Wild dataset. The most
significant gains are for arousal prediction, for which motion information is
intuitively more informative. Source code is available at
this https URL

本文提出了一种称为 MIMAMO（Micro-Macro-Motion）网络的双流递归网络，将微动作和宏动作特征相结合，以改进视频情感识别，并在两个视频情感数据集上实现了最先进的性能。