Table pretrain-then-finetune paradigm has been proposed and employed at a
rapid pace after the success of pre-training in the natural language domain.
Despite the promising findings in tabular pre-trained language models (TPLMs),
there is an input gap between pre-training and fine-tuning phases. For
instance, TPLMs jointly pre-trained with table and text input could be
effective for tasks also with table-text joint input like table question
answering, but it may fail for tasks with only tables or text as input such as
table retrieval. To this end, we propose UTP, an approach that dynamically
supports three types of multi-modal inputs: table-text, table, and text.
Specifically, UTP is pre-trained with two strategies: (1) We first utilize a
universal mask language modeling objective on each kind of input, enforcing the
model to adapt various inputs. (2) We then present Cross-Modal Contrastive
Regularization (CMCR), which utilizes contrastive learning to encourage the
consistency between table-text cross-modality representations via unsupervised
instance-wise training signals during pre-training. By these means, the
resulting model not only bridges the input gap between pre-training and
fine-tuning but also advances in the alignment of table and text. Extensive
results show UTP achieves superior results on uni-modal input tasks (e.g.,
table retrieval) and cross-modal input tasks (e.g., table question answering).

该论文提出了 UTP 这一基于对比学习及多模态输入的数据驱动表格文本预训练方法，可用于单模态和交叉模态输入任务，有效弥合了预训练和微调阶段的输入差距并提高了表和文本的对齐精度。