The transferability of deep neural networks (DNNs) has made significant
progress in image and language processing. However, due to the heterogeneity
among tables, such DNN bonus is still far from being well exploited on tabular
data prediction (e.g., regression or classification tasks). Condensing
knowledge from diverse domains, language models (LMs) possess the capability to
comprehend feature names from various tables, potentially serving as versatile
learners in transferring knowledge across distinct tables and diverse
prediction tasks, but their discrete text representation space is inherently
incompatible with numerical feature values in tables. In this paper, we present
TP-BERTa, a specifically pre-trained LM model for tabular data prediction.
Concretely, a novel relative magnitude tokenization converts scalar numerical
feature values to finely discrete, high-dimensional tokens, and an
intra-feature attention approach integrates feature values with the
corresponding feature names. Comprehensive experiments demonstrate that our
pre-trained TP-BERTa leads the performance among tabular DNNs and is
competitive with Gradient Boosted Decision Tree models in typical tabular data
regime.

TP-BERTa 是一种专门用于表格数据预测的预训练语言模型，通过将数值特征值转换为离散的高维标记，并使用内部特征注意方法将特征值与相应的特征名称相结合，TP-BERTa 在典型的表格数据领域中在表格 DNN 中表现卓越，并与梯度提升决策树模型具备竞争力。