We propose an audio-visual spatial-temporal deep neural network with: (1) a
visual block containing a pretrained 2D-CNN followed by a temporal
convolutional network (TCN); (2) an aural block containing several parallel
TCNs; and (3) a leader-follower attentive fusion block combining the
audio-visual information. The TCN with large history coverage enables our model
to exploit spatial-temporal information within a much larger window length
(i.e., 300) than that from the baseline and state-of-the-art methods (i.e., 36
or 48). The fusion block emphasizes the visual modality while exploits the
noisy aural modality using the inter-modality attention mechanism. To make full
use of the data and alleviate over-fitting, cross-validation is carried out on
the training and validation set. The concordance correlation coefficient (CCC)
centering is used to merge the results from each fold. On the test (validation)
set of the Aff-Wild2 database, the achieved CCC is 0.463 (0.469) for valence
and 0.492 (0.649) for arousal, which significantly outperforms the baseline
method with the corresponding CCC of 0.200 (0.210) and 0.190 (0.230) for
valence and arousal, respectively. The code is available at
this https URL

本文提出一种视听空间时域深度神经网络，其中包括预训练的二维 - CNN 和几个平行 TCNs，同时利用视听信息，通过跨验证来充分利用数据和缓解过度拟合，并使用交界面关注机制来强调视觉模态和利用嘈杂的听觉模态。在测试集中，本文的方法在 valence 和 arousal 方面通过了 CCC 测试，相较于其他方法有明显的准确度提升。