Recent work in speech-to-speech translation (S2ST) has focused primarily on
offline settings, where the full input utterance is available before any output
is given. This, however, is not reasonable in many real-world scenarios. In
latency-sensitive applications, rather than waiting for the full utterance,
translations should be spoken as soon as the information in the input is
present. In this work, we introduce a system for simultaneous S2ST targeting
real-world use cases. Our system supports translation from 57 languages to
English with tunable parameters for dynamically adjusting the latency of the
output -- including four policies for determining when to speak an output
sequence. We show that these policies achieve offline-level accuracy with
minimal increases in latency over a Greedy (wait-$k$) baseline. We open-source
our evaluation code and interactive test script to aid future SimulS2ST
research and application development.

本文提出了一种面向实际应用场景的实时语音翻译系统 SimulS2ST，支持 57 种语言对英语的翻译，具备实时调节输出延迟的功能，能够非常快速地进行语音翻译。

学习何时发声：离线模型下同时语音翻译中的延迟和质量权衡

Learning When to Speak: Latency and Quality Trade-offs for Simultaneous  Speech-to-Speech Translation with Offline Models

Large pre-trained multilingual models like mBERT, XLM-R achieve state of the
art results on language understanding tasks. However, they are not well suited
for latency critical applications on both servers and edge devices. It's
important to reduce the memory and compute resources required by these models.
To this end, we propose pQRNN, a projection-based embedding-free neural encoder
that is tiny and effective for natural language processing tasks. Without
pre-training, pQRNNs significantly outperform LSTM models with pre-trained
embeddings despite being 140x smaller. With the same number of parameters, they
outperform transformer baselines thereby showcasing their parameter efficiency.
Additionally, we show that pQRNNs are effective student architectures for
distilling large pre-trained language models. We perform careful ablations
which study the effect of pQRNN parameters, data augmentation, and distillation
settings. On MTOP, a challenging multilingual semantic parsing dataset, pQRNN
students achieve 95.9\% of the performance of an mBERT teacher while being 350x
smaller. On mATIS, a popular parsing task, pQRNN students on average are able
to get to 97.1\% of the teacher while again being 350x smaller. Our strong
results suggest that our approach is great for latency-sensitive applications
while being able to leverage large mBERT-like models.

介绍了 pQRNN，这是一种基于投影的嵌入式神经编码器，非常适合自然语言处理任务。通过深入研究其参数、数据增强和蒸馏设置等因素，结果表明 pQRNN 对于延迟敏感型应用非常适合，同时可以利用大型 mBERT 等预训练模型。