While a streaming voice assistant system has been used in many applications,
this system typically focuses on unnatural, one-shot interactions assuming
input from a single voice query without hesitation or disfluency. However, a
common conversational utterance often involves multiple queries with
turn-taking, in addition to disfluencies. These disfluencies include pausing to
think, hesitations, word lengthening, filled pauses and repeated phrases. This
makes doing speech recognition with conversational speech, including one with
multiple queries, a challenging task. To better model the conversational
interaction, it is critical to discriminate disfluencies and end of query in
order to allow the user to hold the floor for disfluencies while having the
system respond as quickly as possible when the user has finished speaking. In
this paper, we present a turntaking predictor built on top of the end-to-end
(E2E) speech recognizer. Our best system is obtained by jointly optimizing for
ASR task and detecting when the user is paused to think or finished speaking.
The proposed approach demonstrates over 97% recall rate and 85% precision rate
on predicting true turn-taking with only 100 ms latency on a test set designed
with 4 types of disfluencies inserted in conversational utterances.

本论文提出了一个基于端对端语音识别器的取向预测器，通过检测用户何时停顿思考或讲完话来联合优化 ASR 任务和取向预测任务，以更好地建模对话交互。测试集表明，该方法在识别包含四类语言障碍的会话句子时，具有 97% 以上的准确率和 100 毫秒的延迟。