We provide the first exploration of text-to-text transformers (T5) sentence embeddings. Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model. Our encoder-only models outperforms BERT-based sentence embeddings on both transfer tasks and semantic textual similarity (STS). Our encoder-decoder method achieves further improvement on STS. Scaling up T5 from millions to billions of parameters is found to produce consistent improvements on downstream tasks. Finally, we introduce a two-stage contrastive learning approach that achieves a new state-of-art on STS using sentence embeddings, outperforming both Sentence BERT and SimCSE.

本研究利用文本到文本转换器（T5）提供了对句子嵌入的首次探索。通过3种提取T5句子嵌入的方法，本研究建立了一个新的句子表示转移基准SentGLUE，并在语义文本相似性（STS）等方面超过了Sentence-BERT和SimCSE句子嵌入模型。

来自预训练文本到文本模型的可扩展句子编码器