This paper explores privacy-compliant group-level emotion recognition
''in-the-wild'' within the EmotiW Challenge 2023. Group-level emotion
recognition can be useful in many fields including social robotics,
conversational agents, e-coaching and learning analytics. This research imposes
itself using only global features avoiding individual ones, i.e. all features
that can be used to identify or track people in videos (facial landmarks, body
poses, audio diarization, etc.). The proposed multimodal model is composed of a
video and an audio branches with a cross-attention between modalities. The
video branch is based on a fine-tuned ViT architecture. The audio branch
extracts Mel-spectrograms and feed them through CNN blocks into a transformer
encoder. Our training paradigm includes a generated synthetic dataset to
increase the sensitivity of our model on facial expression within the image in
a data-driven way. The extensive experiments show the significance of our
methodology. Our privacy-compliant proposal performs fairly on the EmotiW
challenge, with 79.24% and 75.13% of accuracy respectively on validation and
test set for the best models. Noticeably, our findings highlight that it is
possible to reach this accuracy level with privacy-compliant features using
only 5 frames uniformly distributed on the video.

本研究在 EmotiW Challenge 2023 中探索了符合隐私规范的野外群体情绪识别。提出的多模态模型通过视频和音频分支以及跨模态注意力实现，使用全局特征而避免使用个体特征，仅依靠分布在视频中的 5 个均匀帧即可达到相当高的准确率。

采用隐私合规特征进行野外多模态群体情绪识别

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant  Features

The task of the emotion recognition in the wild (EmotiW) Challenge is to
assign one of seven emotions to short video clips extracted from Hollywood
style movies. The videos depict acted-out emotions under realistic conditions
with a large degree of variation in attributes such as pose and illumination,
making it worthwhile to explore approaches which consider combinations of
features from multiple modalities for label assignment. In this paper we
present our approach to learning several specialist models using deep learning
techniques, each focusing on one modality. Among these are a convolutional
neural network, focusing on capturing visual information in detected faces, a
deep belief net focusing on the representation of the audio stream, a K-Means
based "bag-of-mouths" model, which extracts visual features around the mouth
region and a relational autoencoder, which addresses spatio-temporal aspects of
videos. We explore multiple methods for the combination of cues from these
modalities into one common classifier. This achieves a considerably greater
accuracy than predictions from our strongest single-modality classifier. Our
method was the winning submission in the 2013 EmotiW challenge and achieved a
test set accuracy of 47.67% on the 2014 dataset.

本研究使用深度学习技术，以多模态方式，使用面部检测、音频流等模态，探索影片情感识别模型，成果在 2013 年的 EmotiW 挑战中成为获胜者，并在 2014 年的数据集上实现了约 47.67% 的准确率。