Automatic speech recognition (ASR) systems can suffer from poor recall for
various reasons, such as noisy audio, lack of sufficient training data, etc.
Previous work has shown that recall can be improved by retrieving rewrite
candidates from a large database of likely, contextually-relevant alternatives
to the hypothesis text using nearest-neighbors search over embeddings of the
ASR hypothesis text to correct and candidate corrections.
However, ASR-hypothesis-based retrieval can yield poor precision if the
textual hypotheses are too phonetically dissimilar to the transcript truth. In
this paper, we eliminate the hypothesis-audio mismatch problem by querying the
correction database directly using embeddings derived from the utterance audio;
the embeddings of the utterance audio and candidate corrections are produced by
multimodal speech-text embedding networks trained to place the embedding of the
audio of an utterance and the embedding of its corresponding textual transcript
close together.
After locating an appropriate correction candidate using nearest-neighbor
search, we score the candidate with its speech-text embedding distance before
adding the candidate to the original n-best list.
We show a relative word error rate (WER) reduction of 6% on utterances whose
transcripts appear in the candidate set, without increasing WER on general
utterances.

通过使用多模式语音文本嵌入网络，从语音文本数据库中直接查询校正候选项，以消除音频 - 假设不匹配问题，并在将候选项添加到原始候选列表之前使用语音 - 文本嵌入距离对候选项进行评分，该方法在改善检索率的同时，降低了相对的词错误率（WER）。