Recent works in end-to-end speech-to-text translation (ST) have proposed
multi-tasking methods with soft parameter sharing which leverage machine
translation (MT) data via secondary encoders that map text inputs to an
eventual cross-modal representation. In this work, we instead propose a ST/MT
multi-tasking framework with hard parameter sharing in which all model
parameters are shared cross-modally. Our method reduces the speech-text
modality gap via a pre-processing stage which converts speech and text inputs
into two discrete token sequences of similar length -- this allows models to
indiscriminately process both modalities simply using a joint vocabulary. With
experiments on MuST-C, we demonstrate that our multi-tasking framework improves
attentional encoder-decoder, Connectionist Temporal Classification (CTC),
transducer, and joint CTC/attention models by an average of +0.5 BLEU without
any external MT data. Further, we show that this framework incorporates
external MT data, yielding +0.8 BLEU, and also improves transfer learning from
pre-trained textual models, yielding +1.8 BLEU.

最近的研究在端到端语音到文本翻译（ST）中提出了多任务方法，通过辅助编码器将机器翻译（MT）数据映射到最终的跨模态表示中，采用软参数共享。本研究提出了一种具有硬参数共享的 ST/MT 多任务框架，通过预处理阶段将语音和文本输入转换为两个长度相似的离散令牌序列，从而减少了语音 - 文本模态差异。通过在 MuST-C 上的实验，我们证明了我们的多任务框架平均提高了 0.5 BLEU 的关注编码器 - 解码器、连接时序分类（CTC）、转录器和联合 CTC / 关注模型，而无需外部 MT 数据。此外，我们还展示了这个框架的融入外部 MT 数据可以产生 0.8 BLEU 的改进，并且可以提高从预训练的文本模型进行迁移学习，从而产生 1.8 BLEU 的改进。

跨模态多任务语音转文字翻译与参数共享

Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard  Parameter Sharing

Multi-Task Learning is a learning paradigm that uses correlated tasks to
improve performance generalization. A common way to learn multiple tasks is
through the hard parameter sharing approach, in which a single architecture is
used to share the same subset of parameters, creating an inductive bias between
them during the training process. Due to its simplicity, potential to improve
generalization, and reduce computational cost, it has gained the attention of
the scientific and industrial communities. However, tasks often conflict with
each other, which makes it challenging to define how the gradients of multiple
tasks should be combined to allow simultaneous learning. To address this
problem, we use the idea of multi-objective optimization to propose a method
that takes into account temporal behaviour of the gradients to create a dynamic
bias that adjust the importance of each task during the backpropagation. The
result of this method is to give more attention to the tasks that are diverging
or that are not being benefited during the last iterations, allowing to ensure
that the simultaneous learning is heading to the performance maximization of
all tasks. As a result, we empirically show that the proposed method
outperforms the state-of-art approaches on learning conflicting tasks. Unlike
the adopted baselines, our method ensures that all tasks reach good
generalization performances.

采用多目标优化的方法，考虑梯度的时间行为，创建一个动态偏差，利用多任务学习的相关性来提高性能和泛化性能，并确保所有任务达到良好的泛化性能。