Video-based Question Answering (Video QA) is a challenging task and becomes
even more intricate when addressing Socially Intelligent Question Answering
(SIQA). SIQA requires context understanding, temporal reasoning, and the
integration of multimodal information, but in addition, it requires processing
nuanced human behavior. Furthermore, the complexities involved are exacerbated
by the dominance of the primary modality (text) over the others. Thus, there is
a need to help the task's secondary modalities to work in tandem with the
primary modality. In this work, we introduce a cross-modal alignment and
subsequent representation fusion approach that achieves state-of-the-art
results (82.06\% accuracy) on the Social IQ 2.0 dataset for SIQA. Our approach
exhibits an improved ability to leverage the video modality by using the audio
modality as a bridge with the language modality. This leads to enhanced
performance by reducing the prevalent issue of language overfitting and
resultant video modality bypassing encountered by current existing techniques.
Our code and models are publicly available at
this https URL

利用跨模态对齐和表示融合方法，在 Social IQ 2.0 数据集上取得了 82.06％的准确率，增强了视频模态的利用能力，并通过降低语言过拟合和当前技术所遇到的视频模态绕过等问题，提高了性能。

听力先行：具有讲话者关注的视频对齐

Listen Then See: Video Alignment with Speaker Attention

We propose a scalable approach to learn video-based question answering (QA):
answer a "free-form natural language question" about a video content. Our
approach automatically harvests a large number of videos and descriptions
freely available online. Then, a large number of candidate QA pairs are
automatically generated from descriptions rather than manually annotated. Next,
we use these candidate QA pairs to train a number of video-based QA methods
extended fromMN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et
al. 2015), SS (Venugopalan et al. 2015). In order to handle non-perfect
candidate QA pairs, we propose a self-paced learning procedure to iteratively
identify them and mitigate their effects in training. Finally, we evaluate
performance on manually generated video-based QA pairs. The results show that
our self-paced learning procedure is effective, and the extended SS model
outperforms various baselines.

本文提出了一种可扩展的视频问答技术，利用自动生成的大量候选问题 - 答案对并使用自适应学习方法以处理其中存在的非理想数据，取得了优于基线模型的效果。