To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as
well as much more unpaired text data by multi-modal training, one needs to
address two problems: 1) the synchronicity of feature sampling rates between
speech and language (aka text data); 2) the homogeneity of the learned
representations from two encoders. In this paper we propose to employ a novel
bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder
(bottom layers) and text encoder with a multi-modal learning method. The BiAM
is to facilitate feature sampling rate exchange, realizing the quality of the
transformed features for the one kind to be measured in another space, with
diversified objective functions. As a result, the speech representations are
enriched with more linguistic information, while the representations generated
by the text encoder are more similar to corresponding speech ones, and
therefore the shared ASR models are more amenable for unpaired text data
pretraining. To validate the efficacy of the proposed method, we perform two
categories of experiments with or without extra unpaired text data.
Experimental results on Librispeech corpus show it can achieve up to 6.15% word
error rate reduction (WERR) with only paired data learning, while 9.23% WERR
when more unpaired text data is employed.

采用双向注意力机制的多模态学习方法，能够有效地提升语音表示的语言信息，增强文本的语音表示，从而使共享的 ASR 模型更适用于无配对的文本数据预训练，仅使用配对数据学习时，单词错误率减少了 6.15％，使用更多无配对文本数据时，错误率减少了 9.23％。

基于双向注意力的语音文本多模态训练，提高语音识别能力

Speech-text based multi-modal training with bidirectional attention for improved speech recognition

Attention-based recurrent neural encoder-decoder models present an elegant
solution to the automatic speech recognition problem. This approach folds the
acoustic model, pronunciation model, and language model into a single network
and requires only a parallel corpus of speech and text for training. However,
unlike in conventional approaches that combine separate acoustic and language
models, it is not clear how to use additional (unpaired) text. While there has
been previous work on methods addressing this problem, a thorough comparison
among methods is still lacking. In this paper, we compare a suite of past
methods and some of our own proposed methods for using unpaired text data to
improve encoder-decoder models. For evaluation, we use the medium-sized
Switchboard data set and the large-scale Google voice search and dictation data
sets. Our results confirm the benefits of using unpaired text across a range of
methods and data sets. Surprisingly, for first-pass decoding, the rather simple
approach of shallow fusion performs best across data sets. However, for Google
data sets we find that cold fusion has a lower oracle error rate and
outperforms other approaches after second-pass rescoring on the Google voice
search data set.

本文通过对一系列过去的方法以及我们提出的某些方法进行比较，探讨如何使用未配对的文本数据来改进编码器 - 解码器模型以及方法的效果和数据集之间的关联。同时结果证实，使用未配对的文本跨越各种方法和数据集具有显著的益处，而 “浅层融合” 的方法在一些数据集上的表现优于其他方法，但在谷歌数据集上，“冷融合” 方法则表现更佳。