Transformer-based neural networks, empowered by Self-Supervised Learning
(SSL), have demonstrated unprecedented performance across various domains.
However, related literature suggests that tabular Transformers may struggle to
outperform classical Machine Learning algorithms, such as Gradient Boosted
Decision Trees (GBDT). In this paper, we aim to challenge GBDTs with tabular
Transformers on a typical task faced in e-commerce, namely fraud detection. Our
study is additionally motivated by the problem of selection bias, often
occurring in real-life fraud detection systems. It is caused by the production
system affecting which subset of traffic becomes labeled. This issue is
typically addressed by sampling randomly a small part of the whole production
data, referred to as a Control Group. This subset follows a target distribution
of production data and therefore is usually preferred for training
classification models with standard ML algorithms. Our methodology leverages
the capabilities of Transformers to learn transferable representations using
all available data by means of SSL, giving it an advantage over classical
methods. Furthermore, we conduct large-scale experiments, pre-training tabular
Transformers on vast amounts of data instances and fine-tuning them on smaller
target datasets. The proposed approach outperforms heavily tuned GBDTs by a
considerable margin of the Average Precision (AP) score. Pre-trained models
show more consistent performance than the ones trained from scratch when
fine-tuning data is limited. Moreover, they require noticeably less labeled
data for reaching performance comparable to their GBDT competitor that utilizes
the whole dataset.

使用 SSL 技术的基于 Transformer 的神经网络在欺诈检测任务中超越 GBDT 算法，在大规模实验中，预训练的 Transformer 模型在有限的微调数据上表现更加一致且需要较少的标记数据。

应用表格转换器挑战 Booking.com 诈骗检测中的梯度提升决策树

Challenging Gradient Boosted Decision Trees with Tabular Transformers  for Fraud Detection at Booking.com

Machine learning methods for estimating heterogeneous treatment effects (HTE)
facilitate large-scale personalized decision-making across various domains such
as healthcare, policy making, education, and more. Current machine learning
approaches for HTE require access to substantial amounts of data per treatment,
and the high costs associated with interventions makes centrally collecting so
much data for each intervention a formidable challenge. To overcome this
obstacle, in this work, we propose a novel framework for collaborative learning
of HTE estimators across institutions via Federated Learning. We show that even
under a diversity of interventions and subject populations across clients, one
can jointly learn a common feature representation, while concurrently and
privately learning the specific predictive functions for outcomes under
distinct interventions across institutions. Our framework and the associated
algorithm are based on this insight, and leverage tabular transformers to map
multiple input data to feature representations which are then used for outcome
prediction via multi-task learning. We also propose a novel way of federated
training of personalised transformers that can work with heterogeneous input
feature spaces. Experimental results on real-world clinical trial data
demonstrate the effectiveness of our method.

使用机器学习方法估计异质治疗效应（HTE）可以在各个领域（如医疗保健、政策制定、教育等）中实现大规模个性化决策。本文提出了一种通过联邦学习在机构之间进行协作学习 HTE 估计器的新框架，实现了在各个机构之间同时学习共同特征表示和不同干预措施下的特定预测函数。我们的框架还通过多任务学习利用表格转换器将多个输入数据映射为特征表示来进行结果预测，并提出了一种适用于异质输入特征空间的个性化转换器的联邦训练方法。实验结果表明了我们的方法的有效性。

用于估计异质治疗效果的联邦学习

Federated Learning for Estimating Heterogeneous Treatment Effects

The success of self-supervised learning in computer vision and natural
language processing has motivated pretraining methods on tabular data. However,
most existing tabular self-supervised learning models fail to leverage
information across multiple data tables and cannot generalize to new tables. In
this work, we introduce XTab, a framework for cross-table pretraining of
tabular transformers on datasets from various domains. We address the challenge
of inconsistent column types and quantities among tables by utilizing
independent featurizers and using federated learning to pretrain the shared
component. Tested on 84 tabular prediction tasks from the OpenML-AutoML
Benchmark (AMLB), we show that (1) XTab consistently boosts the
generalizability, learning speed, and performance of multiple tabular
transformers, (2) by pretraining FT-Transformer via XTab, we achieve superior
performance than other state-of-the-art tabular deep learning models on various
tasks such as regression, binary, and multiclass classification.

本研究提出了一个名为 XTab 的跨表格预训练框架，使用自监督学习算法来提高多种数据集上的表格变换器的泛化性、学习速度和性能，并通过联邦学习解决了跨表不一致的挑战。