Massively multilingual pretrained transformers (MMTs) have tremendously
pushed the state of the art on multilingual NLP and cross-lingual transfer of
NLP models in particular. While a large body of work leveraged MMTs to mine
parallel data and induce bilingual document embeddings, much less effort has
been devoted to training general-purpose (massively) multilingual document
encoder that can be used for both supervised and unsupervised document-level
tasks. In this work, we pretrain a massively multilingual document encoder as a
hierarchical transformer model (HMDE) in which a shallow document transformer
contextualizes sentence representations produced by a state-of-the-art
pretrained multilingual sentence encoder. We leverage Wikipedia as a readily
available source of comparable documents for creating training data, and train
HMDE by means of a cross-lingual contrastive objective, further exploiting the
category hierarchy of Wikipedia for creation of difficult negatives. We
evaluate the effectiveness of HMDE in two arguably most common and prominent
cross-lingual document-level tasks: (1) cross-lingual transfer for topical
document classification and (2) cross-lingual document retrieval. HMDE is
significantly more effective than (i) aggregations of segment-based
representations and (ii) multilingual Longformer. Crucially, owing to its
massively multilingual lower transformer, HMDE successfully generalizes to
languages unseen in document-level pretraining. We publicly release our code
and models at
this https URL .

本研究使用维基百科作为训练数据，通过交叉语言对比的目标来训练具有上下文感知的浅层文档转换器，它可以用于监督和非监督文档级任务。该模型在两个常见的跨语言文档级任务中比基于段落的表示和多语言 Longformer 表现更好，并成功推广到了未见过的语言上。