Most existing speech disfluency detection techniques only rely upon acoustic
data. In this work, we present a practical multimodal disfluency detection
approach that leverages available video data together with audio. We curate an
audiovisual dataset and propose a novel fusion technique with unified
weight-sharing modality-agnostic encoders to learn the temporal and semantic
context. Our resilient design accommodates real-world scenarios where the video
modality may sometimes be missing during inference. We also present alternative
fusion strategies when both modalities are assured to be complete. In
experiments across five disfluency-detection tasks, our unified multimodal
approach significantly outperforms Audio-only unimodal methods, yielding an
average absolute improvement of 10% (i.e., 10 percentage point increase) when
both video and audio modalities are always available, and 7% even when video
modality is missing in half of the samples.

现有的语音故障检测技术只依赖于声学数据，本研究提出了一种实用的多模态故障检测方法，利用可用的视频数据与音频结合。我们策划了一个音频 - 视觉数据集，并提出了一种新颖的融合技术，使用权值共享的模态不可知编码器来学习时态和语义上下文。我们的可靠设计适应了视频模态在推断过程中有时可能丢失的现实场景。当两种模态始终可用时，我们还提供了备选融合策略。在五个故障检测任务的实验中，我们的统一多模态方法明显优于仅音频的单模态方法，当视频和音频模态始终可用时，平均绝对改进率为 10%（即百分点增加 10%），即使在一半的样本中视频模态丢失，仍然有 7% 的改进。