End-to-end speech summarization (E2E SSum) is a technique to directly
generate summary sentences from speech. Compared with the cascade approach,
which combines automatic speech recognition (ASR) and text summarization
models, the E2E approach is more promising because it mitigates ASR errors,
incorporates nonverbal information, and simplifies the overall system. However,
since collecting a large amount of paired data (i.e., speech and summary) is
difficult, the training data is usually insufficient to train a robust E2E SSum
system. In this paper, we present two novel methods that leverage a large
amount of external text summarization data for E2E SSum training. The first
technique is to utilize a text-to-speech (TTS) system to generate synthesized
speech, which is used for E2E SSum training with the text summary. The second
is a TTS-free method that directly inputs phoneme sequence instead of
synthesized speech to the E2E SSum model. Experiments show that our proposed
TTS- and phoneme-based methods improve several metrics on the How2 dataset. In
particular, our best system outperforms a previous state-of-the-art one by a
large margin (i.e., METEOR score improvements of more than 6 points). To the
best of our knowledge, this is the first work to use external language
resources for E2E SSum. Moreover, we report a detailed analysis of the How2
dataset to confirm the validity of our proposed E2E SSum system.

本文介绍了两种新颖的方法，利用大量外部文本摘要数据为 E2E SSum 训练提供支持，分别是使用 TTS 系统生成合成语音进行训练，以及使用输入音素序列而非合成语音直接训练模型。实验证明，这些方法在 How2 数据集上改进了多个指标，并且是第一篇使用外部语言资源进行 E2E SSum 的工作。