Classical machine learning models, such as linear models and tree-based
models, are widely used in industry. These models are sensitive to data
distribution, thus feature preprocessing, which transforms features from one
distribution to another, is a crucial step to ensure good model quality.
Manually constructing a feature preprocessing pipeline is challenging because
data scientists need to make difficult decisions about which preprocessors to
select and in which order to compose them. In this paper, we study how to
automate feature preprocessing (Auto-FP) for tabular data. Due to the large
search space, a brute-force solution is prohibitively expensive. To address
this challenge, we interestingly observe that Auto-FP can be modelled as either
a hyperparameter optimization (HPO) or a neural architecture search (NAS)
problem. This observation enables us to extend a variety of HPO and NAS
algorithms to solve the Auto-FP problem. We conduct a comprehensive evaluation
and analysis of 15 algorithms on 45 public ML datasets. Overall,
evolution-based algorithms show the leading average ranking. Surprisingly, the
random search turns out to be a strong baseline. Many surrogate-model-based and
bandit-based search algorithms, which achieve good performance for HPO and NAS,
do not outperform random search for Auto-FP. We analyze the reasons for our
findings and conduct a bottleneck analysis to identify the opportunities to
improve these algorithms. Furthermore, we explore how to extend Auto-FP to
support parameter search and compare two ways to achieve this goal. In the end,
we evaluate Auto-FP in an AutoML context and discuss the limitations of popular
AutoML tools. To the best of our knowledge, this is the first study on
automated feature preprocessing. We hope our work can inspire researchers to
develop new algorithms tailored for Auto-FP.

通过分析 15 种算法在 45 个公开的机器学习数据集上的综合评估和分析，研究表明 Auto-FP 问题可以用超参数优化 (HPO) 或神经架构搜索 (NAS) 问题来建模，演化算法在平均排名上表现最好，随机搜索是一个强有力的基准线，许多基于代理模型和赌博算法搜索的算法并没有超过随机搜索在 Auto-FP 方面的性能。

自动 FP：针对表格数据的自动特征预处理的实验研究

Auto-FP: An Experimental Study of Automated Feature Preprocessing for  Tabular Data

Feature preprocessing continues to play a critical role when applying machine
learning and statistical methods to tabular data. In this paper, we propose the
use of the kernel density integral transformation as a feature preprocessing
step. Our approach subsumes the two leading feature preprocessing methods as
limiting cases: linear min-max scaling and quantile transformation. We
demonstrate that, without hyperparameter tuning, the kernel density integral
transformation can be used as a simple drop-in replacement for either method,
offering robustness to the weaknesses of each. Alternatively, with tuning of a
single continuous hyperparameter, we frequently outperform both of these
methods. Finally, we show that the kernel density transformation can be
profitably applied to statistical data analysis, particularly in correlation
analysis and univariate clustering.

该论文提出使用核密度积分变换作为特征预处理步骤，可以替代传统的特征预处理方法，并在机器学习和统计方法中表现出更好的鲁棒性和性能。

核密度积分变换

The Kernel Density Integral Transformation

Training machine learning models with differential privacy (DP) has received
increasing interest in recent years. One of the most popular algorithms for
training differentially private models is differentially private stochastic
gradient descent (DPSGD) and its variants, where at each step gradients are
clipped and combined with some noise. Given the increasing usage of DPSGD, we
ask the question: is DPSGD alone sufficient to find a good minimizer for every
dataset under privacy constraints? As a first step towards answering this
question, we show that even for the simple case of linear classification,
unlike non-private optimization, (private) feature preprocessing is vital for
differentially private optimization. In detail, we first show theoretically
that there exists an example where without feature preprocessing, DPSGD incurs
a privacy error proportional to the maximum norm of features over all samples.
We then propose an algorithm called DPSGD-F, which combines DPSGD with feature
preprocessing and prove that for classification tasks, it incurs a privacy
error proportional to the diameter of the features $\max_{x, x' \in D} \|x -
x'\|_2$. We then demonstrate the practicality of our algorithm on image
classification benchmarks.

用差分隐私训练机器学习模型的方法在近年来受到越来越多的关注。我们通过研究线性分类的简单案例，首次展示出在差分隐私优化中，除了使用差分隐私随机梯度下降算法（DPSGD）外，特征预处理对于优化结果的重要性。我们提出了一种名为 DPSGD-F 的算法，结合了 DPSGD 和特征预处理，并证明了对于分类任务，它的隐私误差与特征的直径成正比。我们还通过图像分类基准测试展示了我们算法的实用性。