It has been shown that learning audiovisual features can lead to improved
speech recognition performance over audio-only features, especially for noisy
speech. However, in many common applications, the visual features are partially
or entirely missing, e.g.~the speaker might move off screen. Multi-modal models
need to be robust: missing video frames should not degrade the performance of
an audiovisual model to be worse than that of a single-modality audio-only
model. While there have been many attempts at building robust models, there is
little consensus on how robustness should be evaluated. To address this, we
introduce a framework that allows claims about robustness to be evaluated in a
precise and testable way. We also conduct a systematic empirical study of the
robustness of common audiovisual speech recognition architectures on a range of
acoustic noise conditions and test suites. Finally, we show that an
architecture-agnostic solution based on cascades can consistently achieve
robustness to missing video, even in settings where existing techniques for
robustness like dropout fall short.

通过引入一个可以准确且可测试地评估鲁棒性的框架，我们对常见的多模态语音识别架构在各种噪声条件和测试套件中的鲁棒性进行了系统的实证研究，并展示了一种基于级联的与架构无关的解决方案，可以在存在缺失视频的情况下持续实现鲁棒性。

关于音视觉语音识别对缺失视频的鲁棒性

On Robustness to Missing Video for Audiovisual Speech Recognition

Active speaker detection (ASD) in videos with multiple speakers is a
challenging task as it requires learning effective audiovisual features and
spatial-temporal correlations over long temporal windows. In this paper, we
present SPELL, a novel spatial-temporal graph learning framework that can solve
complex tasks such as ASD. To this end, each person in a video frame is first
encoded in a unique node for that frame. Nodes corresponding to a single person
across frames are connected to encode their temporal dynamics. Nodes within a
frame are also connected to encode inter-person relationships. Thus, SPELL
reduces ASD to a node classification task. Importantly, SPELL is able to reason
over long temporal contexts for all nodes without relying on computationally
expensive fully connected graph neural networks. Through extensive experiments
on the AVA-ActiveSpeaker dataset, we demonstrate that learning graph-based
representations can significantly improve the active speaker detection
performance owing to its explicit spatial and temporal structure. SPELL
outperforms all previous state-of-the-art approaches while requiring
significantly lower memory and computational resources. Our code is publicly
available at this https URL

本文提出了 SPELL，一种新的空间时序图学习框架，通过对每个视频帧中的人员建立节点、建立连接来将复杂任务如活跃扬声器检测转化为节点分类任务，实现了对所有节点的长时间上下文推理，显式地利用了空间和时间结构，明显提高了检测性能，且所需的计算和内存资源显著地降低。