Relational tables on the Web store a vast amount of knowledge. Owing to the
wealth of such tables, there has been tremendous progress on a variety of tasks
in the area of table understanding. However, existing work generally relies on
heavily-engineered task-specific features and model architectures. In this
paper, we present TURL, a novel framework that introduces the
pre-training/fine-tuning paradigm to relational Web tables. During
pre-training, our framework learns deep contextualized representations on
relational tables in an unsupervised manner. Its universal model design with
pre-trained representations can be applied to a wide range of tasks with
minimal task-specific fine-tuning. Specifically, we propose a structure-aware
Transformer encoder to model the row-column structure of relational tables, and
present a new Masked Entity Recovery (MER) objective for pre-training to
capture the semantics and knowledge in large-scale unlabeled data. We
systematically evaluate TURL with a benchmark consisting of 6 different tasks
for table understanding (e.g., relation extraction, cell filling). We show that
TURL generalizes well to all tasks and substantially outperforms existing
methods in almost all instances.

本文提出了 TURL 框架，该框架利用预训练 / 微调方法，在无监督的情况下学习关系表的深度上下文化表示，并应用于各种任务中。通过结构感知的 Transformer 编码器建模关系表的行列结构，并引入了一种新的 Masked Entity Recovery 目标进行预训练。在 6 个不同的关系表理解任务中，TURL 广泛适用且相对于现有方法具有更好的表现。