Embeddings play an important role in end-to-end solutions for multi-modal
language processing problems. Although there has been some effort to understand
the properties of single-modality embedding spaces, particularly that of text,
their cross-modal counterparts are less understood. In this work, we study some
intrinsic properties of a joint speech-text embedding space, constructed by
minimizing the distance between paired utterance and transcription inputs in a
teacher-student model setup, that are informative for several prominent use
cases. We found that incorporating automatic speech recognition through both
pretraining and multitask scenarios aid semantic alignment significantly,
resulting in more tightly coupled embeddings. To analyse cross-modal embeddings
we utilise a quantitative retrieval accuracy metric for semantic alignment,
zero-shot classification for generalisability, and probing of the encoders to
observe the extent of knowledge transfer from one modality to another.

本论文研究联合语音 - 文本 Embeddings 空间的内在属性，借助自动语音识别，通过多任务预训练场景实现语义对齐，利用定量检索精度度量语义对齐，进行了深入分析。