There has been an increasing interest in large speech models that can perform
multiple speech processing tasks in a single model. Such models usually adopt
the encoder-decoder or decoder-only architecture due to their popularity and
good performance in many domains. However, autoregressive models can be slower
during inference compared to non-autoregressive models and also have potential
risks of hallucination. Though prior studies observed promising results of
non-autoregressive models for certain tasks at small scales, it remains unclear
if they can be scaled to speech-to-text generation in diverse languages and
tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we
propose OWSM-CTC, a novel encoder-only speech foundation model based on
Connectionist Temporal Classification (CTC). It is trained on 180k hours of
public audio data for multilingual automatic speech recognition (ASR), speech
translation (ST), and language identification (LID). Compared to
encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up
to 25% relative improvement on ST, while it is more robust and 3 to 4 times
faster for inference. OWSM-CTC also improves the long-form ASR result with 20x
speed-up. We will publicly release our codebase, pre-trained model, and
training logs to promote open science in speech foundation models.

基于连接主义时间分类（CTC）的新型仅编码器语音模型（OWSM-CTC）在多语言自动语音识别（ASR），语音翻译（ST）和语言识别（LID）任务上取得了有竞争力的结果，并在 ST 上提高了 25％的相对改进，在推断中更为稳健且速度更快，同时也对长形式 ASR 结果有 20 倍的加速。