We propose a novel approach to optimizing a byte-level representation for
end-to-end automatic speech recognition (ASR). Byte-level representation is
often used by large scale multilingual ASR systems when the character set of
the supported languages is large. The compactness and universality of
byte-level representation allow the ASR models to use smaller output
vocabularies and therefore, provide more flexibility. UTF-8 is a commonly used
byte-level representation for multilingual ASR, but it is not designed to
optimize machine learning tasks directly. By using auto-encoder and vector
quantization, we show that we can optimize a byte-level representation for ASR
and achieve better accuracy. Our proposed framework can incorporate information
from different modalities, and provides an error correction mechanism. In an
English/Mandarin dictation task, we show that a bilingual ASR model built with
this approach can outperform UTF-8 representation by 5% relative in error rate.

我们提出了一种对字节级表示进行优化的新方法，通过使用自编码器和向量量化，在自动语音识别中实现更好的准确性。

优化端到端自动语音识别的字节级表示

Optimizing Byte-level Representation for End-to-end ASR

We present two end-to-end models: Audio-to-Byte (A2B) and Byte-to-Audio
(B2A), for multilingual speech recognition and synthesis. Prior work has
predominantly used characters, sub-words or words as the unit of choice to
model text. These units are difficult to scale to languages with large
vocabularies, particularly in the case of multilingual processing. In this
work, we model text via a sequence of Unicode bytes, specifically, the UTF-8
variable length byte sequence for each character. Bytes allow us to avoid large
softmaxes in languages with large vocabularies, and share representations in
multilingual models. We show that bytes are superior to grapheme characters
over a wide variety of languages in monolingual end-to-end speech recognition.
Additionally, our multilingual byte model outperform each respective single
language baseline on average by 4.4% relatively. In Japanese-English
code-switching speech, our multilingual byte model outperform our monolingual
baseline by 38.6% relatively. Finally, we present an end-to-end multilingual
speech synthesis model using byte representations which matches the performance
of our monolingual baselines.

本文提出了两个端到端模型：Audio-to-Byte (A2B) 和 Byte-to-Audio (B2A)，用于多语言语音识别和合成。模型的关键是使用 Unicode 字节序列来建模文本，并且通过使用字节代替大的 softmax 函数，不同语言之间可以共享表示。同时通过引入字节表示，作者在单语言和多语言模型的表现中都获得了很好的结果。