End-to-end speech summarization (E2E SSum) directly summarizes input speech
into easy-to-read short sentences with a single model. This approach is
promising because it, in contrast to the conventional cascade approach, can
utilize full acoustical information and mitigate to the propagation of
transcription errors. However, due to the high cost of collecting
speech-summary pairs, an E2E SSum model tends to suffer from training data
scarcity and output unnatural sentences. To overcome this drawback, we propose
for the first time to integrate a pre-trained language model (LM), which is
highly capable of generating natural sentences, into the E2E SSum decoder via
transfer learning. In addition, to reduce the gap between the independently
pre-trained encoder and decoder, we also propose to transfer the baseline E2E
SSum encoder instead of the commonly used automatic speech recognition encoder.
Experimental results show that the proposed model outperforms baseline and data
augmented models.

本文提出在端对端语音摘要（E2E SSum）模型中集成预训练语言模型来解决数据稀缺问题，并通过迁移学习来减少编解码器之间的差距。实验证明，该模型表现优于基线和数据增强模型。

预训练语言模型迁移学习改进端到端语音摘要

Transfer Learning from Pre-trained Language Models Improves End-to-End  Speech Summarization

End-to-end speech summarization (E2E SSum) is a technique to directly
generate summary sentences from speech. Compared with the cascade approach,
which combines automatic speech recognition (ASR) and text summarization
models, the E2E approach is more promising because it mitigates ASR errors,
incorporates nonverbal information, and simplifies the overall system. However,
since collecting a large amount of paired data (i.e., speech and summary) is
difficult, the training data is usually insufficient to train a robust E2E SSum
system. In this paper, we present two novel methods that leverage a large
amount of external text summarization data for E2E SSum training. The first
technique is to utilize a text-to-speech (TTS) system to generate synthesized
speech, which is used for E2E SSum training with the text summary. The second
is a TTS-free method that directly inputs phoneme sequence instead of
synthesized speech to the E2E SSum model. Experiments show that our proposed
TTS- and phoneme-based methods improve several metrics on the How2 dataset. In
particular, our best system outperforms a previous state-of-the-art one by a
large margin (i.e., METEOR score improvements of more than 6 points). To the
best of our knowledge, this is the first work to use external language
resources for E2E SSum. Moreover, we report a detailed analysis of the How2
dataset to confirm the validity of our proposed E2E SSum system.

本文介绍了两种新颖的方法，利用大量外部文本摘要数据为 E2E SSum 训练提供支持，分别是使用 TTS 系统生成合成语音进行训练，以及使用输入音素序列而非合成语音直接训练模型。实验证明，这些方法在 How2 数据集上改进了多个指标，并且是第一篇使用外部语言资源进行 E2E SSum 的工作。