Feature selection is the problem of selecting a subset of features for a
machine learning model that maximizes model quality subject to a budget
constraint. For neural networks, prior methods, including those based on
$\ell_1$ regularization, attention, and other techniques, typically select the
entire feature subset in one evaluation round, ignoring the residual value of
features during selection, i.e., the marginal contribution of a feature given
that other features have already been selected. We propose a feature selection
algorithm called Sequential Attention that achieves state-of-the-art empirical
results for neural networks. This algorithm is based on an efficient one-pass
implementation of greedy forward selection and uses attention weights at each
step as a proxy for feature importance. We give theoretical insights into our
algorithm for linear regression by showing that an adaptation to this setting
is equivalent to the classical Orthogonal Matching Pursuit (OMP) algorithm, and
thus inherits all of its provable guarantees. Our theoretical and empirical
analyses offer new explanations towards the effectiveness of attention and its
connections to overparameterization, which may be of independent interest.

该研究提出一种名为 Sequential Attention 的特征选择算法，使用注意力权重作为特征重要性的代理，实现了神经网络的最新实证结果；通过在线性回归方面提出的理论分析，可与经典的 Orthogonal Matching Pursuit (OMP) 算法等效，并继承了所有的可证明的保证；理论和经验分析为注意力的有效性及其与过度参数化的联系提供了新的解释。

特征选择的序列注意力

Sequential Attention for Feature Selection

We propose a novel high-performance and interpretable canonical deep tabular
data learning architecture, TabNet. TabNet uses sequential attention to choose
which features to reason from at each decision step, enabling interpretability
and more efficient learning as the learning capacity is used for the most
salient features. We demonstrate that TabNet outperforms other neural network
and decision tree variants on a wide range of non-performance-saturated tabular
datasets and yields interpretable feature attributions plus insights into the
global model behavior. Finally, for the first time to our knowledge, we
demonstrate self-supervised learning for tabular data, significantly improving
performance with unsupervised representation learning when unlabeled data is
abundant.

本研究提出了一种新的、高性能和可解释的深度表格数据学习架构 TabNet，采用顺序注意方法在每个决策步骤中选择推理特征，实现了可解释性和更高效的学习。我们证明，在各种非性能饱和表格数据集上，TabNet 胜过其他神经网络和决策树变体，并提供了可解释的特征归因和对全局模型行为的深入认识。最后，我们首次在表格数据上展示了自监督学习，在未标记的数据丰富情况下显著提高了性能。