Training a high performance end-to-end speech (E2E) processing model requires
an enormous amount of labeled speech data, especially in the era of
data-centric artificial intelligence. However, labeled speech data are usually
scarcer and more expensive for collection, compared to textual data. We propose
Latent Synthesis (LaSyn), an efficient textual data utilization framework for
E2E speech processing models. We train a latent synthesizer to convert textual
data into an intermediate latent representation of a pre-trained speech model.
These pseudo acoustic representations of textual data augment acoustic data for
model training. We evaluate LaSyn on low-resource automatic speech recognition
(ASR) and spoken language understanding (SLU) tasks. For ASR, LaSyn improves an
E2E baseline trained on LibriSpeech train-clean-100, with relative word error
rate reductions over 22.3% on different test sets. For SLU, LaSyn improves our
E2E baseline by absolute 4.1% for intent classification accuracy and 3.8% for
slot filling SLU-F1 on SLURP, and absolute 4.49% and 2.25% for exact match (EM)
and EM-Tree accuracies on STOP respectively. With fewer parameters, the results
of LaSyn are competitive to published state-of-the-art works. The results
demonstrate the quality of the augmented training data. The source code will be
available to the community.

通过训练一个转换文本数据的潜在合成器，我们提出了一种高效的文本数据利用框架 Latent Synthesis (LaSyn)，用于 E2E 语音处理模型。在低资源自动语音识别（ASR）和口语理解（SLU）任务中，LaSyn 通过为模型训练提供了伪声学表示的文本数据，分别在不同测试集上使 E2E 基线的词错误率降低了 22.3% 以上，并提高了意图分类准确率 4.1% 和槽位填充准确率 3.8%，在 STOP 上分别提高了 4.49% 和 2.25% 的准确率。同时，LaSyn 使用更少的参数，取得了与现有最先进作品相媲美的结果，证明了增强训练数据的质量。