We present our work on developing a multilingual, efficient text-to-text
transformer that is suitable for handling long inputs. This model, called
mLongT5, builds upon the architecture of LongT5, while leveraging the
multilingual datasets used for pretraining mT5 and the pretraining tasks of
UL2. We evaluate this model on a variety of multilingual summarization and
question-answering tasks, and the results show stronger performance for mLongT5
when compared to existing multilingual models such as mBART or M-BERT.

本文介绍了我们关于开发一种适用于处理长输入的多语言高效文本到文本转换器的工作，称为 mLongT5 (即在 LongT5 架构上构建并利用用于预训练 mT5 和 UL2 预训练任务的多语言数据集)，我们评估了该模型在各种多语言摘要和问答任务上的表现，并与现有的多语言模型（如 mBART 或 M-BERT）进行了比较，表明 mLongT5 相对于这些模型具有更强的性能。

mLongT5：针对更长序列的多语言高效文本转换器

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for  Longer Sequences

Recent work has shown that either (1) increasing the input length or (2)
increasing model size can improve the performance of Transformer-based neural
models. In this paper, we present a new model, called LongT5, with which we
explore the effects of scaling both the input length and model size at the same
time. Specifically, we integrated attention ideas from long-input transformers
(ETC), and adopted pre-training strategies from summarization pre-training
(PEGASUS) into the scalable T5 architecture. The result is a new attention
mechanism we call {\em Transient Global} (TGlobal), which mimics ETC's
local/global attention mechanism, but without requiring additional side-inputs.
We are able to achieve state-of-the-art results on several summarization tasks
and outperform the original T5 models on question answering tasks.

本文介绍了一种新模型 LongT5，该模型将长输入变换（ETC）的注意力机制和摘要预训练（PEGASUS）的预训练策略整合到可扩展的 T5 架构中，取得了更好的自然语言摘要和问答系统结果。