In end-to-end speech translation, speech and text pre-trained models improve
translation quality. Recently proposed models simply connect the pre-trained
models of speech and text as encoder and decoder. Therefore, only the
information from the final layer of encoders is input to the decoder. Since it
is clear that the speech pre-trained model outputs different information from
each layer, the simple connection method cannot fully utilize the information
that the speech pre-trained model has. In this study, we propose an
inter-connection mechanism that aggregates the information from each layer of
the speech pre-trained model by weighted sums and inputs into the decoder. This
mechanism increased BLEU by approximately 2 points in en-de, en-ja, and en-zh
by increasing parameters by 2K when the speech pre-trained model was frozen.
Furthermore, we investigated the contribution of each layer for each language
by visualizing layer weights and found that the contributions were different.

本研究提出了一种互连机制，通过加权求和来聚合语言预训练模型的每一层信息并输入到解码器中，从而提高了 BLEU 大约 2 分，在冻结语言预训练模型时加入了 2K 的参数。此外，我们通过可视化层权重，发现每个语言的每层的贡献是不同的。