Recently, Transformer-like deep architectures have shown strong performance
on tabular data problems. Unlike traditional models, e.g., MLP, these
architectures map scalar values of numerical features to high-dimensional
embeddings before mixing them in the main backbone. In this work, we argue that
embeddings for numerical features are an underexplored degree of freedom in
tabular DL, which allows constructing more powerful DL models and competing
with GBDT on some traditionally GBDT-friendly benchmarks. We start by
describing two conceptually different approaches to building embedding modules:
the first one is based on a piecewise linear encoding of scalar values, and the
second one utilizes periodic activations. Then, we empirically demonstrate that
these two approaches can lead to significant performance boosts compared to the
embeddings based on conventional blocks such as linear layers and ReLU
activations. Importantly, we also show that embedding numerical features is
beneficial for many backbones, not only for Transformers. Specifically, after
proper embeddings, simple MLP-like models can perform on par with the
attention-based architectures. Overall, we highlight embeddings for numerical
features as an important design aspect with good potential for further
improvements in tabular DL.

本研究针对数值特征在计算表格数据方面的表现，提出在深度学习中使用数值特征嵌入提升性能的方法，并重点研究两种不同的嵌入方式，并实现了嵌入在简单 MLP-like 模型与 attention-based 结构中的对比性能提升。