It has been shown that learning audiovisual features can lead to improved
speech recognition performance over audio-only features, especially for noisy
speech. However, in many common applications, the visual features are partially
or entirely missing, e.g.~the speaker might move off screen. Multi-modal models
need to be robust: missing video frames should not degrade the performance of
an audiovisual model to be worse than that of a single-modality audio-only
model. While there have been many attempts at building robust models, there is
little consensus on how robustness should be evaluated. To address this, we
introduce a framework that allows claims about robustness to be evaluated in a
precise and testable way. We also conduct a systematic empirical study of the
robustness of common audiovisual speech recognition architectures on a range of
acoustic noise conditions and test suites. Finally, we show that an
architecture-agnostic solution based on cascades can consistently achieve
robustness to missing video, even in settings where existing techniques for
robustness like dropout fall short.

通过引入一个可以准确且可测试地评估鲁棒性的框架，我们对常见的多模态语音识别架构在各种噪声条件和测试套件中的鲁棒性进行了系统的实证研究，并展示了一种基于级联的与架构无关的解决方案，可以在存在缺失视频的情况下持续实现鲁棒性。