This paper presents techniques and findings for improving the performance of
low-resource speech to text translation (ST). We conducted experiments on both
simulated and real-low resource setups, on language pairs English - Portuguese,
and Tamasheq - French respectively. Using the encoder-decoder framework for ST,
our results show that a multilingual automatic speech recognition system acts
as a good initialization under low-resource scenarios. Furthermore, using the
CTC as an additional objective for translation during training and decoding
helps to reorder the internal representations and improves the final
translation. Through our experiments, we try to identify various factors
(initializations, objectives, and hyper-parameters) that contribute the most
for improvements in low-resource setups. With only 300 hours of pre-training
data, our model achieved 7.3 BLEU score on Tamasheq - French data,
outperforming prior published works from IWSLT 2022 by 1.6 points.

本文提出了改进低资源语音转文本翻译性能的技术和发现，实验结果显示多语言自动语音识别系统在低资源场景下作为良好的初始化，CTC 作为额外的目标有助于重新排序内部表示并提高最终翻译。通过实验，确定了对改进低资源设置最有贡献的各种因素（初始化、目标和超参数）。在只有 300 小时的预训练数据下，模型在 Tamasheq - French 数据中取得了 7.3 的 BLEU 分数，比 IWSLT 2022 的先前发表作品高 1.6 个百分点。

基于预训练 ASR 模型的低资源语音转文本翻译策略

Strategies for improving low resource speech to text translation relying  on pre-trained ASR models

Pretraining and multitask learning are widely used to improve the speech to
text translation performance. In this study, we are interested in training a
speech to text translation model along with an auxiliary text to text
translation task. We conduct a detailed analysis to understand the impact of
the auxiliary task on the primary task within the multitask learning framework.
Our analysis confirms that multitask learning tends to generate similar decoder
representations from different modalities and preserve more information from
the pretrained text translation modules. We observe minimal negative transfer
effect between the two tasks and sharing more parameters is helpful to transfer
knowledge from the text task to the speech task. The analysis also reveals that
the modality representation difference at the top decoder layers is still not
negligible, and those layers are critical for the translation quality. Inspired
by these findings, we propose three methods to improve translation quality.
First, a parameter sharing and initialization strategy is proposed to enhance
information sharing between the tasks. Second, a novel attention-based
regularization is proposed for the encoders and pulls the representations from
different modalities closer. Third, an online knowledge distillation is
proposed to enhance the knowledge transfer from the text to the speech task.
Our experiments show that the proposed approach improves translation
performance by more than 2 BLEU over a strong baseline and achieves
state-of-the-art results on the \textsc{MuST-C} English-German, English-French
and English-Spanish language pairs.

本文研究了在多任务学习框架下，辅助文本翻译任务对语音到文本翻译任务的影响及参数共享等解决方案，提出三种方法以提高翻译质量，实验结果表明该方法在几种语种翻译任务中相对于基线都达到了最新水平。