Disfluency detection has mainly been solved in a pipeline approach, as
post-processing of speech recognition. In this study, we propose
Transformer-based encoder-decoder models that jointly solve speech recognition
and disfluency detection, which work in a streaming manner. Compared to
pipeline approaches, the joint models can leverage acoustic information that
makes disfluency detection robust to recognition errors and provide non-verbal
clues. Moreover, joint modeling results in low-latency and lightweight
inference. We investigate two joint model variants for streaming disfluency
detection: a transcript-enriched model and a multi-task model. The
transcript-enriched model is trained on text with special tags indicating the
starting and ending points of the disfluent part. However, it has problems with
latency and standard language model adaptation, which arise from the additional
disfluency tags. We propose a multi-task model to solve such problems, which
has two output layers at the Transformer decoder; one for speech recognition
and the other for disfluency detection. It is modeled to be conditioned on the
currently recognized token with an additional token-dependency mechanism. We
show that the proposed joint models outperformed a BERT-based pipeline approach
in both accuracy and latency, on both the Switchboard and the corpus of
spontaneous Japanese.

本研究通过基于 Transformer 框架的编码器 - 解码器模型提出了一种联合解决语音识别和流式识别中断相关的自然语言分析难题的方法，将其与传统的流水线工艺相比较，表现出更优异的准确性和实时性。