Recently, speech-text pre-training methods have shown remarkable success in
many speech and natural language processing tasks. However, most previous
pre-trained models are usually tailored for one or two specific tasks, but fail
to conquer a wide range of speech-text tasks. In addition, existing speech-text
pre-training methods fail to explore the contextual information within a
dialogue to enrich utterance representations. In this paper, we propose
Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT
cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog
pre-training model. Concretely, to consider the temporality of speech modality,
we design a novel temporal position prediction task to capture the speech-text
alignment. This pre-training task aims to predict the start and end time of
each textual word in the corresponding speech waveform. In addition, to learn
the characteristics of spoken dialogs, we generalize a response selection task
from textual dialog pre-training to speech-text dialog pre-training scenarios.
Experimental results on four different downstream speech-text tasks demonstrate
the superiority of SPECTRA in learning speech-text alignment and multi-turn
dialog context.

本文提出了第一种语音 - 文本对话预训练模型 SPECTRA，在输入上进行了一些优化，如引入时间信息，增加上下文理解能力，并通过实验进行了充分验证，展示了 SPECTRA 模型在语音 - 文本对话中学习言语 - 文本对齐和多轮对话上下文的能力比其他模型更强。

口语对话理解的语音 - 文本对话预训练与显式跨模态对齐

Speech-Text Dialog Pre-training for Spoken Dialog Understanding with  Explicit Cross-Modal Alignment

Stickers with vivid and engaging expressions are becoming increasingly
popular in online messaging apps, and some works are dedicated to automatically
select sticker response by matching text labels of stickers with previous
utterances. However, due to their large quantities, it is impractical to
require text labels for the all stickers. Hence, in this paper, we propose to
recommend an appropriate sticker to user based on multi-turn dialog context
history without any external labels. Two main challenges are confronted in this
task. One is to learn semantic meaning of stickers without corresponding text
labels. Another challenge is to jointly model the candidate sticker with the
multi-turn dialog context. To tackle these challenges, we propose a sticker
response selector (SRS) model. Specifically, SRS first employs a convolutional
based sticker image encoder and a self-attention based multi-turn dialog
encoder to obtain the representation of stickers and utterances. Next, deep
interaction network is proposed to conduct deep matching between the sticker
with each utterance in the dialog history. SRS then learns the short-term and
long-term dependency between all interaction results by a fusion network to
output the the final matching score. To evaluate our proposed method, we
collect a large-scale real-world dialog dataset with stickers from one of the
most popular online chatting platform. Extensive experiments conducted on this
dataset show that our model achieves the state-of-the-art performance for all
commonly-used metrics. Experiments also verify the effectiveness of each
component of SRS. To facilitate further research in sticker selection field, we
release this dataset of 340K multi-turn dialog and sticker pairs.

本研究提出了一种基于多轮对话历史上下文的贴纸响应选择器（SRS）模型，无需外部标签即可推荐适当的贴纸，并使用卷积神经网络、自注意力网络和交互网络等技术实现了对话历史中贴纸的表示和匹配，实验结果表明该模型在常用指标上达到了最先进的性能。