Existing work on tabular representation learning jointly models tables and
associated text using self-supervised objective functions derived from
pretrained language models such as BERT. While this joint pretraining improves
tasks involving paired tables and text (e.g., answering questions about
tables), we show that it underperforms on tasks that operate over tables
without any associated text (e.g., populating missing cells). We devise a
simple pretraining objective (corrupt cell detection) that learns exclusively
from tabular data and reaches the state-of-the-art on a suite of table based
prediction tasks. Unlike competing approaches, our model (TABBIE) provides
embeddings of all table substructures (cells, rows, and columns), and it also
requires far less compute to train. A qualitative analysis of our model's
learned cell, column, and row representations shows that it understands complex
table semantics and numerical trends.

通过自我学习目标函数和预训练语言模型（如 BERT）对表格和相关文本进行联合建模的现有工作可以提高涉及配对表格和文本的任务（例如回答关于表格的问题）的性能，但在没有任何相关文本的情况下处理表格的任务（例如填充缺失的单元格）时表现不佳。我们提出了一种简单的预训练目标（损坏单元格检测），该预训练目标只是从表格数据中学习，并且在一套基于表格的预测任务上达到了最先进的水平，从而超过了竞争方法。与其他方法不同，我们的模型（TABBIE）提供了所有表子结构（单元格、行和列）的嵌入，并且所需的计算时间也要少得多。我们对模型的学习单元格、列和行的表示进行了质量分析，结果表明它可以理解复杂的表语义和数字趋势。