Deep Learning has revolutionized the field of AI and led to remarkable
achievements in applications involving image and text data. Unfortunately,
there is inconclusive evidence on the merits of neural networks for structured
tabular data. In this paper, we introduce a large-scale empirical study
comparing neural networks against gradient-boosted decision trees on tabular
data, but also transformer-based architectures against traditional multi-layer
perceptrons (MLP) with residual connections. In contrast to prior work, our
empirical findings indicate that neural networks are competitive against
decision trees. Furthermore, we assess that transformer-based architectures do
not outperform simpler variants of traditional MLP architectures on tabular
datasets. As a result, this paper helps the research and practitioner
communities make informed choices on deploying neural networks on future
tabular data applications.

通过对大规模的实证研究，本文发现神经网络在结构化表格数据上与决策树具有竞争力，而基于变压器的架构在表格数据集上并不能超过传统 MLP 架构的简化版本。这些发现有助于研究和实践社区在未来的表格数据应用中做出明智的选择。

表格数据：关注力是您所需的全部吗？

Tabular Data: Is Attention All You Need?

Deep learning (DL) models for tabular data problems are receiving
increasingly more attention, while the algorithms based on gradient-boosted
decision trees (GBDT) remain a strong go-to solution. Following the recent
trends in other domains, such as natural language processing and computer
vision, several retrieval-augmented tabular DL models have been recently
proposed. For a given target object, a retrieval-based model retrieves other
relevant objects, such as the nearest neighbors, from the available (training)
data and uses their features or even labels to make a better prediction.
However, we show that the existing retrieval-based tabular DL solutions provide
only minor, if any, benefits over the properly tuned simple retrieval-free
baselines. Thus, it remains unclear whether the retrieval-based approach is a
worthy direction for tabular DL.
In this work, we give a strong positive answer to this question. We start by
incrementally augmenting a simple feed-forward architecture with an
attention-like retrieval component similar to those of many (tabular)
retrieval-based models. Then, we highlight several details of the attention
mechanism that turn out to have a massive impact on the performance on tabular
data problems, but that were not explored in prior work. As a result, we design
TabR -- a simple retrieval-based tabular DL model which, on a set of public
benchmarks, demonstrates the best average performance among tabular DL models,
becomes the new state-of-the-art on several datasets, and even outperforms GBDT
models on the recently proposed ``GBDT-friendly'' benchmark (see the first
figure).

通过逐步增加一个与诸多表格检索模型类似的注意力检索组件，TabR 模型在一组公开基准测试中表现出最佳平均性能，成为几个数据集上的最新技术标准，甚至在最近提出的 “GBDT 友好” 基准测试中超越了 GBDT 模型。

TabR: 发掘检索辅助的表格深度学习的能力

TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning

Tabular data is one of the most commonly used types of data in machine
learning. Despite recent advances in neural nets (NNs) for tabular data, there
is still an active discussion on whether or not NNs generally outperform
gradient-boosted decision trees (GBDTs) on tabular data, with several recent
works arguing either that GBDTs consistently outperform NNs on tabular data, or
vice versa. In this work, we take a step back and ask, 'does it matter?' We
conduct the largest tabular data analysis to date, by comparing 19 algorithms
across 176 datasets, and we find that the 'NN vs. GBDT' debate is
overemphasized: for a surprisingly high number of datasets, either the
performance difference between GBDTs and NNs is negligible, or light
hyperparameter tuning on a GBDT is more important than selecting the best
algorithm. Next, we analyze 965 metafeatures to determine what properties of a
dataset make NNs or GBDTs better-suited to perform well. For example, we find
that GBDTs are much better than NNs at handling skewed feature distributions,
heavy-tailed feature distributions, and other forms of dataset irregularities.
Our insights act as a guide for practitioners to decide whether or not they
need to run a neural net to reach top performance on their dataset. Our
codebase and all raw results are available at
this https URL

该研究分析了在标签数据上使用神经网络（NN）和梯度增强决策树（GBDT）的性能差异，发现针对不同数据集，NN 和 GBDT 的表现各有千秋，需要根据数据集来选择合适的算法并进行超参数调优。

神经网络何时在表格数据上优于增强树？

When Do Neural Nets Outperform Boosted Trees on Tabular Data?

Gradient-boosted decision trees (GBDT) are widely used and highly effective
machine learning approach for tabular data modeling. However, their complex
structure may lead to low robustness against small covariate perturbation in
unseen data. In this study, we apply one-hot encoding to convert a GBDT model
into a linear framework, through encoding of each tree leaf to one dummy
variable. This allows for the use of linear regression techniques, plus a novel
risk decomposition for assessing the robustness of a GBDT model against
covariate perturbations. We propose to enhance the robustness of GBDT models by
refitting their linear regression forms with $L_1$ or $L_2$ regularization.
Theoretical results are obtained about the effect of regularization on the
model performance and robustness. It is demonstrated through numerical
experiments that the proposed regularization approach can enhance the
robustness of the one-hot-encoded GBDT models.

本文研究使用 one-hot encoding 和 linear regression 以及 L1 或 L2 正则化对 Gradient-boosted decision trees 进行改进，提高其对协变量扰动的容忍能力。通过数值实验和理论分析证明了该方法可以增强模型的鲁棒性。