In large part due to their implicit semantic modeling, self-supervised
learning (SSL) methods have significantly increased the performance of valence
recognition in speech emotion recognition (SER) systems. Yet, their large size
may often hinder practical implementations. In this work, we take HuBERT as an
example of an SSL model and analyze the relevance of each of its layers for
SER. We show that shallow layers are more important for arousal recognition
while deeper layers are more important for valence. This observation motivates
the importance of additional textual information for accurate valence
recognition, as the distilled framework lacks the depth of its large-scale SSL
teacher. Thus, we propose an audio-textual distilled SSL framework that, while
having only ~20% of the trainable parameters of a large SSL model, achieves on
par performance across the three emotion dimensions (arousal, valence,
dominance) on the MSP-Podcast v1.10 dataset.

本文旨在将 HuBERT 等 self-supervised learning 的模型应用在 SER 系统中，并通过分析模型的每一层得出更好的情感识别效果，提出基于 audio-textual distilled SSL 的模型实现更为高效的情感识别。