We explore unifying a neural segmenter with two-pass cascaded encoder ASR
into a single model. A key challenge is allowing the segmenter (which runs in
real-time, synchronously with the decoder) to finalize the 2nd pass (which runs
900 ms behind real-time) without introducing user-perceived latency or deletion
errors during inference. We propose a design where the neural segmenter is
integrated with the causal 1st pass decoder to emit a end-of-segment (EOS)
signal in real-time. The EOS signal is then used to finalize the non-causal 2nd
pass. We experiment with different ways to finalize the 2nd pass, and find that
a novel dummy frame injection strategy allows for simultaneous high quality 2nd
pass results and low finalization latency. On a real-world long-form captioning
task (YouTube), we achieve 2.4% relative WER and 140 ms EOS latency gains over
a baseline VAD-based segmenter with the same cascaded encoder.

我们探讨了将神经分割器与两次级联编码器 ASR 统一为单个模型。我们提出了一种设计，其中神经分割器与因果一次传递解码器集成在一起，以实时发出分段结束（EOS）信号。我们尝试了不同的方式来最终确定第二遍，发现一种新颖的虚拟帧注入策略可以同时实现高质量的第二遍结果和低延迟。在一个实际的长形字幕任务（YouTube）中，我们比基线 VAD 基于分割器使用相同的级联编码器获得了 2.4% 的相对 WER 和 140 毫秒的 EOS 延迟增益。