This work presents a sparse-attention Transformer architecture for modeling
documents that contain large tables. Tables are ubiquitous on the web, and are
rich in information. However, more than 20% of relational tables on the web
have 20 or more rows (Cafarella et al., 2008), and these large tables present a
challenge for current Transformer models, which are typically limited to 512
tokens. Here we propose MATE, a novel Transformer architecture designed to
model the structure of web tables. MATE uses sparse attention in a way that
allows heads to efficiently attend to either rows or columns in a table. This
architecture scales linearly with respect to speed and memory, and can handle
documents containing more than 8000 tokens with current accelerators. MATE also
has a more appropriate inductive bias for tabular data, and sets a new
state-of-the-art for three table reasoning datasets. For HybridQA (Chen et al.,
2020b), a dataset that involves large documents containing tables, we improve
the best prior result by 19 points.

使用稀疏注意力 Transformer 建构的 MATE 结构，可用于处理包含大量表格的网页文档。MATE 能够高效地处理表格的行列信息，并对涵盖超过 8000 个单元格的文档进行建模，同时在处理包含表格的大型文档时也表现出优越性。我们在三个涉及表格的推理数据集上创造了最新的最佳结果，其中包括了大型文档和表格的 HybridQA 数据集，我们的表现提高了 19 分。