Semantic representation learning for sentences is an important and
well-studied problem in NLP. The current trend for this task involves training
a Transformer-based sentence encoder through a contrastive objective with text,
i.e., clustering sentences with semantically similar meanings and scattering
others. In this work, we find the performance of Transformer models as sentence
encoders can be improved by training with multi-modal multi-task losses, using
unpaired examples from another modality (e.g., sentences and unrelated
image/audio data). In particular, besides learning by the contrastive loss on
text, our model clusters examples from a non-linguistic domain (e.g.,
visual/audio) with a similar contrastive loss at the same time. The reliance of
our framework on unpaired non-linguistic data makes it language-agnostic,
enabling it to be widely applicable beyond English NLP. Experiments on 7
semantic textual similarity benchmarks reveal that models trained with the
additional non-linguistic (images/audio) contrastive objective lead to higher
quality sentence embeddings. This indicates that Transformer models are able to
generalize better by doing a similar task (i.e., clustering) with unpaired
examples from different modalities in a multi-task fashion.

本文提出了一种多模态多任务的 Transformer 模型，通过对句子和非语言数据对比学习的方式来提高句子编码器的性能，并在基准测试中获得了更高质量的语义文本相似性结果。