The end-to-end speech translation (E2E-ST) model has gradually become a
mainstream paradigm due to its low latency and less error propagation. However,
it is non-trivial to train such a model well due to the task complexity and
data scarcity. The speech-and-text modality differences result in the E2E-ST
model performance usually inferior to the corresponding machine translation
(MT) model. Based on the above observation, existing methods often use
sharingmechanisms to carry out implicit knowledge transfer by imposing various
constraints. However, the final model often performs worse on the MT task than
the MT model trained alone, which means that the knowledge transfer ability of
this method is also limited. To deal with these problems, we propose the FCCL
(Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which
makes explicit knowledge transfer through cross-modal multi-grained contrastive
learning. A key ingredient of our approach is applying contrastive learning at
both sentence- and frame-level to give the comprehensive guide for extracting
speech representations containing rich semantic information.In addition, we
adopt a simple whitening method to alleviate the representation degeneration in
the MT model, which adversely affects contrast learning. Experiments on the
MuST-C benchmark show that our proposed approach significantly outperforms the
state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis
indicates that FCCL can free up its capacity from learning grammatical
structure information and force more layers to learn semantic information.

该论文提出了一种 FCCL 方法，通过跨模态多粒度对比学习来实现显式知识转移，该方法在多语言和多粒度上显著优于 E2E-ST 基线，有助于提取包含丰富语义信息的语音表示。