Decision trees and their ensembles are endowed with a rich set of diagnostic
tools for ranking and screening variables in a predictive model. Despite the
widespread use of tree based variable importance measures, pinning down their
theoretical properties has been challenging and therefore largely unexplored.
To address this gap between theory and practice, we derive finite sample
performance guarantees for variable selection in nonparametric models using a
single-level CART decision tree (a decision stump). Under standard operating
assumptions in variable screening literature, we find that the marginal signal
strength of each variable and ambient dimensionality can be considerably weaker
and higher, respectively, than state-of-the-art nonparametric variable
selection methods. Furthermore, unlike previous marginal screening methods that
attempt to directly estimate each marginal projection via a truncated basis
expansion, the fitted model used here is a simple, parsimonious decision stump,
thereby eliminating the need for tuning the number of basis terms. Thus,
surprisingly, even though decision stumps are highly inaccurate for estimation
purposes, they can still be used to perform consistent model selection.

通过使用决策树，我们推导出了非参数模型中变量选择的有限样本性能保证，并发现决策树可以用于实现一致的模型选择，即使用于估计目的时高度不准确。

最优决策桩的非参数变量筛选

Nonparametric Variable Screening with Optimal Decision Stumps

This paper reviews and advocates against the use of permute-and-predict (PaP)
methods for interpreting black box functions. Methods such as the variable
importance measures proposed for random forests, partial dependence plots, and
individual conditional expectation plots remain popular because they are both
model-agnostic and depend only on the pre-trained model output, making them
computationally efficient and widely available in software. However, numerous
studies have found that these tools can produce diagnostics that are highly
misleading, particularly when there is strong dependence among features. The
purpose of our work here is to (i) review this growing body of literature, (ii)
provide further demonstrations of these drawbacks along with a detailed
explanation as to why they occur, and (iii) advocate for alternative measures
that involve additional modeling. In particular, we describe how breaking
dependencies between features in hold-out data places undue emphasis on sparse
regions of the feature space by forcing the original model to extrapolate to
regions where there is little to no data. We explore these effects across
various model setups and find support for previous claims in the literature
that PaP metrics can vastly over-emphasize correlated features in both variable
importance measures and partial dependence plots. As an alternative, we discuss
and recommend more direct approaches that involve measuring the change in model
performance after muting the effects of the features under investigation.

本文研究回顾并反对使用 Permuate-and-predict 方法来解释黑盒函数，特别是在特征之间存在强依赖性时，常用的变量重要性度量、局部依赖图和个体条件期望图具有误导性，我们提出了使用额外建模来替代这些指标的建议，并描述了如何在保持数据特征独立性的情况下测量模型性能变化。

无限制置换强制外推：变量重要性需要至少一个更多的模型，或者不存在自由变量重要性

Unrestricted Permutation forces Extrapolation: Variable Importance  Requires at least One More Model, or There Is No Free Variable Importance

Data analysis and machine learning have become an integrative part of the
modern scientific methodology, offering automated procedures for the prediction
of a phenomenon based on past observations, unraveling underlying patterns in
data and providing insights about the problem. Yet, caution should avoid using
machine learning as a black-box tool, but rather consider it as a methodology,
with a rational thought process that is entirely dependent on the problem under
study. In particular, the use of algorithms should ideally require a reasonable
understanding of their mechanisms, properties and limitations, in order to
better apprehend and interpret their results.
Accordingly, the goal of this thesis is to provide an in-depth analysis of
random forests, consistently calling into question each and every part of the
algorithm, in order to shed new light on its learning capabilities, inner
workings and interpretability. The first part of this work studies the
induction of decision trees and the construction of ensembles of randomized
trees, motivating their design and purpose whenever possible. Our contributions
follow with an original complexity analysis of random forests, showing their
good computational performance and scalability, along with an in-depth
discussion of their implementation details, as contributed within Scikit-Learn.
In the second part of this work, we analyse and discuss the interpretability
of random forests in the eyes of variable importance measures. The core of our
contributions rests in the theoretical characterization of the Mean Decrease of
Impurity variable importance measure, from which we prove and derive some of
its properties in the case of multiway totally randomized trees and in
asymptotic conditions. In consequence of this work, our analysis demonstrates
that variable importances [...].

本文对随机森林进行了深入分析，重点讨论了其学习能力、内部运作和可解释性。本文的贡献在于通过理论分析，证明了随机森林的变量重要性可解释性，从而揭示了随机森林在数据分析和机器学习中的应用潜力。