In this paper, we propose a novel architecture for direct extractive
speech-to-speech summarization, ESSumm, which is an unsupervised model without
dependence on intermediate transcribed text. Different from previous methods
with text presentation, we are aimed at generating a summary directly from
speech without transcription. First, a set of smaller speech segments are
extracted based on speech signal's acoustic features. For each candidate speech
segment, a distance-based summarization confidence score is designed for latent
speech representation measure. Specifically, we leverage the off-the-shelf
self-supervised convolutional neural network to extract the deep speech
features from raw audio. Our approach automatically predicts the optimal
sequence of speech segments that capture the key information with a target
summary length. Extensive results on two well-known meeting datasets (AMI and
ICSI corpora) show the effectiveness of our direct speech-based method to
improve the summarization quality with untranscribed data. We also observe that
our unsupervised speech-based method even performs on par with recent
transcript-based summarization approaches, where extra speech recognition is
required.

本文提出了一种新颖的直接抽取式语音摘要架构 ESSumm，它是一种基于深度学习的无线束模型，可以直接从语音中生成摘要，极大地提高了从未转录的数据的摘要质量。