Emotion recognition models using audio input data can enable the development
of interactive systems with applications in mental healthcare, marketing,
gaming, and social media analysis. While the field of affective computing using
audio data is rich, a major barrier to achieve consistently high-performance
models is the paucity of available training labels. Self-supervised learning
(SSL) is a family of methods which can learn despite a scarcity of supervised
labels by predicting properties of the data itself. To understand the utility
of self-supervised learning for audio-based emotion recognition, we have
applied self-supervised learning pre-training to the classification of emotions
from the CMU- MOSEI's acoustic modality. Unlike prior papers that have
experimented with raw acoustic data, our technique has been applied to encoded
acoustic data. Our model is first pretrained to uncover the randomly-masked
timestamps of the acoustic data. The pre-trained model is then fine-tuned using
a small sample of annotated data. The performance of the final model is then
evaluated via several evaluation metrics against a baseline deep learning model
with an identical backbone architecture. We find that self-supervised learning
consistently improves the performance of the model across all metrics. This
work shows the utility of self-supervised learning for affective computing,
demonstrating that self-supervised learning is most useful when the number of
training examples is small, and that the effect is most pronounced for emotions
which are easier to classify such as happy, sad and anger. This work further
demonstrates that self-supervised learning works when applied to embedded
feature representations rather than the traditional approach of pre-training on
the raw input space.

情感识别通过音频输入数据的模型可以实现在心理健康、营销、游戏和社交媒体分析等领域的交互式系统的发展。

基于音频的情绪识别的自监督学习

Self-Supervised Learning for Audio-Based Emotion Recognition

This paper investigates the effectiveness and implementation of
modality-specific large-scale pre-trained encoders for multimodal sentiment
analysis~(MSA). Although the effectiveness of pre-trained encoders in various
fields has been reported, conventional MSA methods employ them for only
linguistic modality, and their application has not been investigated. This
paper compares the features yielded by large-scale pre-trained encoders with
conventional heuristic features. One each of the largest pre-trained encoders
publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for
visual, acoustic, and linguistic modalities, respectively. Experiments on two
datasets reveal that methods with domain-specific pre-trained encoders attain
better performance than those with conventional features in both unimodal and
multimodal scenarios. We also find it better to use the outputs of the
intermediate layers of the encoders than those of the output layer. The codes
are available at this https URL.

本文研究了多模态情感分析中使用具有模态感受的大规模预训练编码器的有效性和实现。通过实验比较了在视觉、声学和语言感受模态下最大的预训练编码器和传统的启发式特征后，得出领域专用预训练编码器比常规特征在单模态和多模态情况下性能更好的结论。