The information in tables can be an important complement to text, making
table-based question answering (QA) systems of great value. The intrinsic
complexity of handling tables often adds an extra burden to both model design
and data annotation. In this paper, we aim to develop a simple table-based QA
model with minimal annotation effort. Motivated by the fact that table-based QA
requires both alignment between questions and tables and the ability to perform
complicated reasoning over multiple table elements, we propose an omnivorous
pretraining approach that consumes both natural and synthetic data to endow
models with these respective abilities. Specifically, given freely available
tables, we leverage retrieval to pair them with relevant natural sentences for
mask-based pretraining, and synthesize NL questions by converting SQL sampled
from tables for pretraining with a QA loss. We perform extensive experiments in
both few-shot and full settings, and the results clearly demonstrate the
superiority of our model OmniTab, with the best multitasking approach achieving
an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively,
also establishing a new state-of-the-art on WikiTableQuestions. Detailed
ablations and analyses reveal different characteristics of natural and
synthetic data, shedding light on future directions in omnivorous pretraining.
Code, pretraining data, and pretrained models are available at
this https URL.

本文旨在开发一种简单的基于表格的问答模型，采用全能预训练方法结合自然和合成数据，实现问题与表格的对齐和多元素复杂推理，实验证明该模型在少量和全量数据上性能卓越，在 WikiTableQuestions 数据集上成为新的最优模型，讨论分析自然数据和合成数据的不同特点，为全能预训练提供未来方向。