In this paper, we present our advanced solutions to the two sub-challenges of
Affective Behavior Analysis in the wild (ABAW) 2023: the Emotional Reaction
Intensity (ERI) Estimation Challenge and Expression (Expr) Classification
Challenge. ABAW 2023 aims to tackle the challenge of affective behavior
analysis in natural contexts, with the ultimate goal of creating intelligent
machines and robots that possess the ability to comprehend human emotions,
feelings, and behaviors. For the Expression Classification Challenge, we
propose a streamlined approach that handles the challenges of classification
effectively. However, our main contribution lies in our use of diverse models
and tools to extract multimodal features such as audio and video cues from the
Hume-Reaction dataset. By studying, analyzing, and combining these features, we
significantly enhance the model's accuracy for sentiment prediction in a
multimodal context. Furthermore, our method achieves outstanding results on the
Emotional Reaction Intensity (ERI) Estimation Challenge, surpassing the
baseline method by an impressive 84\% increase, as measured by the Pearson
Coefficient, on the validation dataset.

本文论述了针对 ABAW2023 中的情感行为分析领域的两个子挑战 - 表情分类和情感反应强度评估的高级解决方案。该方案使用了多种模型和工具来提取多模态特征，并有效地处理分类中的挑战，从而显著提高了模型在多模态情况下的情感预测精度。在情感反应强度评估方面，我们的方法超过了基线方法 84％，并在验证数据集上取得了出色的结果。

基于 Transformer 的视频情感反应强度估计和表达分类的多模态特征提取与融合

Multimodal Feature Extraction and Fusion for Emotional Reaction  Intensity Estimation and Expression Classification in Videos with  Transformers

This paper introduces our method for the Emotional Reaction Intensity (ERI)
Estimation Challenge, in CVPR 2023: 5th Workshop and Competition on Affective
Behavior Analysis in-the-wild (ABAW). Based on the multimodal data provided by
the originazers, we extract acoustic and visual features with different
pretrained models. The multimodal features are mixed together by Transformer
Encoders with cross-modal attention mechnism. In this paper, 1. better features
are extracted with the SOTA pretrained models. 2. Compared with the baseline,
we improve the Pearson's Correlations Coefficient a lot. 3. We process the data
with some special skills to enhance performance ability of our model.

本文介绍了我们在 CVPR 2023：Affective Behavior Analysis in-the-wild (ABAW) 第 5 次研讨会和比赛中进行情感反应强度 (ERI) 估计挑战的方法，通过基于多模态数据提取声学和视觉特征，提高了我们的模型表现能力，并使用 Transformer 编码器进行交叉模态关注机制，从而提高了与基线的 Pearson 相关系数。

基于多模态数据的情绪反应强度估计

Emotional Reaction Intensity Estimation Based on Multimodal Data

Emotional Reaction Intensity(ERI) estimation is an important task in
multimodal scenarios, and has fundamental applications in medicine, safe
driving and other fields. In this paper, we propose a solution to the ERI
challenge of the fifth Affective Behavior Analysis in-the-wild(ABAW), a
dual-branch based multi-output regression model. The spatial attention is used
to better extract visual features, and the Mel-Frequency Cepstral Coefficients
technology extracts acoustic features, and a method named modality dropout is
added to fusion multimodal features. Our method achieves excellent results on
the official validation set.

本文提出了一种解决第五届野外情感行为分析（ABAW）中 Emotional Reaction Intensity（ERI）挑战的双分支基于多输出回归模型，利用空间关注更好地提取视觉特征，利用 Mel-Frequency Cepstral Coefficients 技术提取声学特征，并添加了一种名为模态丢失的融合多模态特征的方法，在官方验证集上取得了优秀的结果。