Recently, speech-text pre-training methods have shown remarkable success in
many speech and natural language processing tasks. However, most previous
pre-trained models are usually tailored for one or two specific tasks, but fail
to conquer a wide range of speech-text tasks. In addition, existing speech-text
pre-training methods fail to explore the contextual information within a
dialogue to enrich utterance representations. In this paper, we propose
Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT
cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog
pre-training model. Concretely, to consider the temporality of speech modality,
we design a novel temporal position prediction task to capture the speech-text
alignment. This pre-training task aims to predict the start and end time of
each textual word in the corresponding speech waveform. In addition, to learn
the characteristics of spoken dialogs, we generalize a response selection task
from textual dialog pre-training to speech-text dialog pre-training scenarios.
Experimental results on four different downstream speech-text tasks demonstrate
the superiority of SPECTRA in learning speech-text alignment and multi-turn
dialog context.

本文提出了第一种语音 - 文本对话预训练模型 SPECTRA，在输入上进行了一些优化，如引入时间信息，增加上下文理解能力，并通过实验进行了充分验证，展示了 SPECTRA 模型在语音 - 文本对话中学习言语 - 文本对齐和多轮对话上下文的能力比其他模型更强。