End-to-end (E2E) speech-to-text translation (ST) often depends on pretraining
its encoder and/or decoder using source transcripts via speech recognition or
text translation tasks, without which translation performance drops
substantially. However, transcripts are not always available, and how
significant such pretraining is for E2E ST has rarely been studied in the
literature. In this paper, we revisit this question and explore the extent to
which the quality of E2E ST trained on speech-translation pairs alone can be
improved. We reexamine several techniques proven beneficial to ST previously,
and offer a set of best practices that biases a Transformer-based E2E ST system
toward training from scratch. Besides, we propose parameterized distance
penalty to facilitate the modeling of locality in the self-attention model for
speech. On four benchmarks covering 23 languages, our experiments show that,
without using any transcripts or pretraining, the proposed system reaches and
even outperforms previous studies adopting pretraining, although the gap
remains in (extremely) low-resource settings. Finally, we discuss neural
acoustic feature modeling, where a neural model is designed to extract acoustic
features from raw speech signals directly, with the goal to simplify inductive
biases and add freedom to the model in describing speech. For the first time,
we demonstrate its feasibility and show encouraging results on ST tasks.

本文通过重新审视机器翻译中的 ST 和文本翻译任务技术，提出了一套训练 E2E ST 系统的最佳实践，其中包括参数化距离惩罚、模拟本地化技术等。实验表明，在不使用转录文本和预训练的情况下，该系统可以达到并甚至超过之前使用预训练方法的研究水平，对于极低资源环境依然有差距。此外，我们还首次证明了神经声学特征建模的可行性，并在 ST 任务中取得了令人鼓舞的效果。

从零开始重新审视端到端的语音到文本翻译

Revisiting End-to-End Speech-to-Text Translation From Scratch

Self-supervised models for speech processing emerged recently as popular
foundation blocks in speech processing pipelines. These models are pre-trained
on unlabeled audio data and then used in speech processing downstream tasks
such as automatic speech recognition (ASR) or speech translation (ST). Since
these models are now used in research and industrial systems alike, it becomes
necessary to understand the impact caused by some features such as gender
distribution within pre-training data. Using French as our investigation
language, we train and compare gender-specific wav2vec 2.0 models against
models containing different degrees of gender balance in their pre-training
data. The comparison is performed by applying these models to two
speech-to-text downstream tasks: ASR and ST. Results show the type of
downstream integration matters. We observe lower overall performance using
gender-specific pre-training before fine-tuning an end-to-end ASR system.
However, when self-supervised models are used as feature extractors, the
overall ASR and ST results follow more complex patterns in which the balanced
pre-trained model does not necessarily lead to the best results. Lastly, our
crude 'fairness' metric, the relative performance difference measured between
female and male test sets, does not display a strong variation from balanced to
gender-specific pre-trained wav2vec 2.0 models.

本研究使用法语作为研究语言，通过比较不同性别平衡的预训练数据集在 ASR 和 ST 中的表现来探究其影响。结果表明，性别平衡的预训练模型不一定导致最佳结果，并且使用自监督模型作为特征提取器时，ASR 和 ST 的结果会遵循更复杂的模式。