Tabular data (or tables) are the most widely used data format in machine
learning (ML). However, ML models often assume the table structure keeps fixed
in training and testing. Before ML modeling, heavy data cleaning is required to
merge disparate tables with different columns. This preprocessing often incurs
significant data waste (e.g., removing unmatched columns and samples). How to
learn ML models from multiple tables with partially overlapping columns? How to
incrementally update ML models as more columns become available over time? Can
we leverage model pretraining on multiple distinct tables? How to train an ML
model which can predict on an unseen table?
To answer all those questions, we propose to relax fixed table structures by
introducing a Transferable Tabular Transformer (TransTab) for tables. The goal
of TransTab is to convert each sample (a row in the table) to a generalizable
embedding vector, and then apply stacked transformers for feature encoding. One
methodology insight is combining column description and table cells as the raw
input to a gated transformer model. The other insight is to introduce
supervised and self-supervised pretraining to improve model performance. We
compare TransTab with multiple baseline methods on diverse benchmark datasets
and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00,
1.78 out of 12 methods in supervised learning, feature incremental learning,
and transfer learning scenarios, respectively; and the proposed pretraining
leads to 2.3% AUC lift on average over the supervised learning.

该研究论文提出了一种名为转移性表格变压器（TransTab）的新方法，该方法通过将表格中的每个样本（即行）转换为可推广的嵌入向量，然后应用堆叠的变压器进行特征编码，来学习来自多个表格的机器学习模型。该方法探讨了如何在训练和测试期间使用多个具有部分重叠列的表格进行机器学习模型的预测，并对预训练模型的性能进行了分析和比较。