End-to-end Speech Translation (ST) aims to convert speech into target text
within a unified model. The inherent differences between speech and text
modalities often impede effective cross-modal and cross-lingual transfer.
Existing methods typically employ hard alignment (H-Align) of individual speech
and text segments, which can degrade textual representations. To address this,
we introduce Soft Alignment (S-Align), using adversarial training to align the
representation spaces of both modalities. S-Align creates a modality-invariant
space while preserving individual modality quality. Experiments on three
languages from the MuST-C dataset show S-Align outperforms H-Align across
multiple tasks and offers translation capabilities on par with specialized
translation models.

使用软对齐方法进行端到端语音翻译，通过对抗训练对齐语音和文本之间的表示空间，提高翻译性能。

端到端语音翻译的模态空间软对齐

Soft Alignment of Modality Space for End-to-end Speech Translation

Despite some empirical success at correcting exposure bias in machine
translation, scheduled sampling algorithms suffer from a major drawback: they
incorrectly assume that words in the reference translations and in sampled
sequences are aligned at each time step. Our new differentiable sampling
algorithm addresses this issue by optimizing the probability that the reference
can be aligned with the sampled output, based on a soft alignment predicted by
the model itself. As a result, the output distribution at each time step is
evaluated with respect to the whole predicted sequence. Experiments on IWSLT
translation tasks show that our approach improves BLEU compared to maximum
likelihood and scheduled sampling baselines. In addition, our approach is
simpler to train with no need for sampling schedule and yields models that
achieve larger improvements with smaller beam sizes.

该研究论文介绍了一种新的可微分采样算法，该算法通过基于模型预测的软对齐来优化参考与采样输出之间的对齐概率，从而提高机器翻译的 BLEU 值，并在 IWSLT 翻译任务中取得较大的改进。