We have described a novel approach for training tabular data using the
TabTransformer model with self-supervised learning. Traditional machine
learning models for tabular data, such as GBDT are being widely used though our
paper examines the effectiveness of the TabTransformer which is a Transformer
based model optimised specifically for tabular data. The TabTransformer
captures intricate relationships and dependencies among features in tabular
data by leveraging the self-attention mechanism of Transformers. We have used a
self-supervised learning approach in this study, where the TabTransformer
learns from unlabelled data by creating surrogate supervised tasks, eliminating
the need for the labelled data. The aim is to find the most effective
TabTransformer model representation of categorical and numerical features. To
address the challenges faced during the construction of various input settings
into the Transformers. Furthermore, a comparative analysis is also been
conducted to examine performance of the TabTransformer model against baseline
models such as MLP and supervised TabTransformer.
The research has presented with a novel approach by creating various variants
of TabTransformer model namely, Binned-TT, Vanilla-MLP-TT, MLP- based-TT which
has helped to increase the effective capturing of the underlying relationship
between various features of the tabular dataset by constructing optimal inputs.
And further we have employed a self-supervised learning approach in the form of
a masking-based unsupervised setting for tabular data. The findings shed light
on the best way to represent categorical and numerical features, emphasizing
the TabTransormer performance when compared to established machine learning
models and other self-supervised learning methods.

使用自监督学习的 TabTransformer 模型，通过描述一种新颖的表格数据训练方法，提高了对分类和数值特征的建模能力。通过对比传统机器学习模型和自监督学习方法，研究结果显示 TabTransformer 在表格数据上的性能优势。

基于表格数据的深度学习：一种自监督方法

Deep Learning with Tabular Data: A Self-supervised Approach

There is an increasing interest in the application of deep learning
architectures to tabular data. One of the state-of-the-art solutions is
TabTransformer which incorporates an attention mechanism to better track
relationships between categorical features and then makes use of a standard MLP
to output its final logits. In this paper we propose multiple modifications to
the original TabTransformer performing better on binary classification tasks
for three separate datasets with more than 1% AUROC gains. Inspired by gated
MLP, linear projections are implemented in the MLP block and multiple
activation functions are tested. We also evaluate the importance of specific
hyper parameters during training.

本文提出了多种改进的 TabTransformer 模型，其中包含注意力机制和多种激活函数以及线性投影，针对三个数据集的二分类任务取得了超过 1% 的 AUROC 收益。此外，我们还评估了训练过程中特定超参数的重要性。

GatedTabTransformer：表格建模的增强深度学习架构

The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling

We propose TabTransformer, a novel deep tabular data modeling architecture
for supervised and semi-supervised learning. The TabTransformer is built upon
self-attention based Transformers. The Transformer layers transform the
embeddings of categorical features into robust contextual embeddings to achieve
higher prediction accuracy. Through extensive experiments on fifteen publicly
available datasets, we show that the TabTransformer outperforms the
state-of-the-art deep learning methods for tabular data by at least 1.0% on
mean AUC, and matches the performance of tree-based ensemble models.
Furthermore, we demonstrate that the contextual embeddings learned from
TabTransformer are highly robust against both missing and noisy data features,
and provide better interpretability. Lastly, for the semi-supervised setting we
develop an unsupervised pre-training procedure to learn data-driven contextual
embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art
methods.

TabTransformer 是一个基于自注意力变换器 (Transformer) 的深度表格数据建模架构，可应用于监督和半监督学习。经过在 15 个公开数据集上的广泛实验，我们展示了 TabTransformer 在表格数据的深度学习方法中的超越和匹配效果。同时，我们演示了从 TabTransformer 学习的环境嵌入具有高度的鲁棒性，可用于噪声和丢失形式的数据特征，提供更好的可解释性。最后，对于半监督场景，我们开发了一种无监督的预训练过程来学习数据驱动的上下文嵌入，使 AUC 平均提升 2.1%。