Leveraging context information is an intuitive idea to improve performance on
conversational automatic speech recognition(ASR). Previous works usually adopt
recognized hypotheses of historical utterances as preceding context, which may
bias the current recognized hypothesis due to the inevitable
historicalrecognition errors. To avoid this problem, we propose an
audio-textual cross-modal representation extractor to learn contextual
representations directly from preceding speech. Specifically, it consists of
two modal-related encoders, extracting high-level latent features from speech
and the corresponding text, and a cross-modal encoder, which aims to learn the
correlation between speech and text. We randomly mask some input tokens and
input sequences of each modality. Then a token-missing or modal-missing
prediction with a modal-level CTC loss on the cross-modal encoder is performed.
Thus, the model captures not only the bi-directional context dependencies in a
specific modality but also relationships between different modalities. Then,
during the training of the conversational ASR system, the extractor will be
frozen to extract the textual representation of preceding speech, while such
representation is used as context fed to the ASR decoder through attention
mechanism. The effectiveness of the proposed approach is validated on several
Mandarin conversation corpora and the highest character error rate (CER)
reduction up to 16% is achieved on the MagicData dataset.

该研究提出了一种音频 - 文本交叉模态表示提取器，通过 attention 机制，将上下文文本表示作为上下文提供给 ASR 解码器，有效地提高了智能语音识别的性能。