Contrastive cross-modal models such as CLIP and CLAP aid various
vision-language (VL) and audio-language (AL) tasks. However, there has been
limited investigation of and improvement in their language encoder, which is
the central component of encoding natural language descriptions of image/audio
into vector representations. We extensively evaluate how unsupervised and
supervised sentence embedding training affect language encoder quality and
cross-modal task performance. In VL pretraining, we found that sentence
embedding training language encoder quality and aids in cross-modal tasks,
improving contrastive VL models such as CyCLIP. In contrast, AL pretraining
benefits less from sentence embedding training, which may result from the
limited amount of pretraining data. We analyze the representation spaces to
understand the strengths of sentence embedding training, and find that it
improves text-space uniformity, at the cost of decreased cross-modal alignment.

对比交叉模态模型如 CLIP 和 CLAP 对视觉 - 语言（VL）和音频 - 语言（AL）任务有所帮助，本篇研究评估了无监督和监督的句子嵌入训练对语言编码器质量和交叉模态任务性能的影响，发现句子嵌入训练有助于提高对比 VL 模型的性能，但在 AL 预训练中效果较少，可能是由于预训练数据量有限所致。通过对表示空间的分析，句子嵌入训练改善了文本空间的均匀性，但降低了交叉模态对齐性。