Multimodal pre-training with text, layout, and image has achieved SOTA
performance for visually-rich document understanding tasks recently, which
demonstrates the great potential for joint learning across different
modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model
for multilingual document understanding, which aims to bridge the language
barriers for visually-rich document understanding. To accurately evaluate
LayoutXLM, we also introduce a multilingual form understanding benchmark
dataset named XFUND, which includes form understanding samples in 7 languages
(Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and
key-value pairs are manually labeled for each language. Experiment results show
that the LayoutXLM model has significantly outperformed the existing SOTA
cross-lingual pre-trained models on the XFUND dataset. The pre-trained
LayoutXLM model and the XFUND dataset are publicly available at
this https URL

本文提出了一种基于 LayoutXLM 的多模态预训练模型，用于多语言文档理解，并在名为 XFUND 的多语言表单理解基准数据集上进行了验证，结果表明 LayoutXLM 模型在跨语言预训练方面优于现有的 SOTA 模型。

LayoutXLM: 多模态预训练用于多语种视觉丰富文档理解

LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich  Document Understanding

Recent studies have demonstrated the overwhelming advantage of cross-lingual
pre-trained models (PTMs), such as multilingual BERT and XLM, on cross-lingual
NLP tasks. However, existing approaches essentially capture the co-occurrence
among tokens through involving the masked language model (MLM) objective with
token-level cross entropy. In this work, we extend these approaches to learn
sentence-level representations and show the effectiveness on cross-lingual
understanding and generation. Specifically, we propose a Hierarchical
Contrastive Learning (HiCTL) method to (1) learn universal representations for
parallel sentences distributed in one or multiple languages and (2) distinguish
the semantically-related words from a shared cross-lingual vocabulary for each
sentence. We conduct evaluations on two challenging cross-lingual tasks, XTREME
and machine translation. Experimental results show that the HiCTL outperforms
the state-of-the-art XLM-R by an absolute gain of 4.2% accuracy on the XTREME
benchmark as well as achieves substantial improvements on both of the
high-resource and low-resource English-to-X translation tasks over strong
baselines.

本文介绍了一种基于 Hierarchical Contrastive Learning 的方法，用于学习句子级别的跨语言表示方法，在 XTREME 任务和机器翻译任务中实验表明，该方法可以显著提高模型准确性。

跨语言学习通用表征

On Learning Universal Representations Across Languages

In this paper, we introduce XGLUE, a new benchmark dataset that can be used
to train large-scale cross-lingual pre-trained models using multilingual and
bilingual corpora and evaluate their performance across a diverse set of
cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in
English for natural language understanding tasks only, XGLUE has two main
advantages: (1) it provides 11 diversified tasks that cover both natural
language understanding and generation scenarios; (2) for each task, it provides
labeled data in multiple languages. We extend a recent cross-lingual
pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and
generation tasks, which is evaluated on XGLUE as a strong baseline. We also
evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for
comparison.

本文介绍了 XGLUE 数据集，它是一个可用于训练大规模跨语言预训练模型和评估其在多样的跨语言任务中表现的新的基准数据集。与 GLUE 不同的是，XGLUE 提供了 11 种多样化的任务来覆盖自然语言理解和生成等问题，同时为每个任务提供了多种语言的标注数据。本文还扩展了最近的跨语言预训练模型 Unicoder，以涵盖自然语言理解和生成任务，并在 XGLUE 上作为强基线进行评估。此外，本文还评估了多语言 BERT、XLM 和 XLM-R 的基础版本（12 层）进行比较。