In this paper, we extensively present our solutions for the MuSe-Stress
sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment
Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the
level of emotional arousal and valence in a time-continuous manner from
audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict
the level of psycho-physiological arousal from a) human annotations fused with
b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals
from the stressed people. The Ulm-TSST dataset which is a novel subset of the
audio-visual textual Ulm-Trier Social Stress dataset that features German
speakers in a Trier Social Stress Test (TSST) induced stress situation is used
in both sub-challenges. For the MuSe-Stress sub-challenge, we highlight our
solutions in three aspects: 1) the audio-visual features and the bio-signal
features are used for emotional state recognition. 2) the Long Short-Term
Memory (LSTM) with the self-attention mechanism is utilized to capture complex
temporal dependencies within the feature sequences. 3) the late fusion strategy
is adopted to further boost the model's recognition performance by exploiting
complementary information scattered across multimodal sequences. Our proposed
model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on
the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge,
we first extract the audio-visual features and the bio-signal features from
multiple modalities. Then, the LSTM module with the self-attention mechanism,
and the Gated Convolutional Neural Networks (GCNN) as well as the LSTM network
are utilized for modeling the complex temporal dependencies in the sequence.
Finally, the late fusion strategy is used. Our proposed method also achieves
CCC of 0.5412 on the test set, which ranks in the top 3.

本研究使用德语说话者的 Ulm-TSST 数据集，通过使用音频 - 视觉记录和生物信号特征来预测情绪唤醒和价值评估的水平，利用 LSTM 和自我注意机制来捕捉其复杂的时序依赖关系，并借助迟到融合策略进一步提高其识别性能。