Feature selection is a key step when dealing with high dimensional data. In
particular, these techniques simplify the process of knowledge discovery from
the data by selecting the most relevant features out of the noisy, redundant
and irrelevant features. A problem that arises in many of these practical
applications is that the outcome of the feature selection algorithm is not
stable. Thus, small variations in the data may yield very different feature
rankings. Assessing the stability of these methods becomes an important issue
in the previously mentioned situations. We propose an information theoretic
approach based on the Jensen Shannon divergence to quantify this robustness.
Unlike other stability measures, this metric is suitable for different
algorithm outcomes: full ranked lists, feature subsets as well as the lesser
studied partial ranked lists. This generalized metric quantifies the difference
among a whole set of lists with the same size, following a probabilistic
approach and being able to give more importance to the disagreements that
appear at the top of the list. Moreover, it possesses desirable properties
including correction for change, upper lower bounds and conditions for a
deterministic selection. We illustrate the use of this stability metric with
data generated in a fully controlled way and compare it with popular metrics
including the Spearmans rank correlation and the Kunchevas index on feature
ranking and selection outcomes, respectively. Additionally, experimental
validation of the proposed approach is carried out on a real-world problem of
food quality assessment showing its potential to quantify stability from
different perspectives.

特征选择是处理高维数据时的关键步骤，本文提出了一种基于 Jensen Shannon 散度的信息论方法来衡量算法的稳定性，该方法能够适用于不同的算法输出结果，并具备校正变化、上下界和确定性选择条件等优良特性。通过在控制条件下生成的数据和与其他流行度量标准的比较，验证了该稳定性度量标准的实用性和优势。在实际问题中，该方法对于食物质量评估具有潜在的稳定性评估能力。

一种评估特征选择和排序算法稳定性的信息论方法

An information theoretic approach to quantify the stability of feature  selection and ranking algorithms

Background: Unstructured and textual data is increasing rapidly and Latent
Dirichlet Allocation (LDA) topic modeling is a popular data analysis methods
for it. Past work suggests that instability of LDA topics may lead to
systematic errors. Aim: We propose a method that relies on replicated LDA runs,
clustering, and providing a stability metric for the topics. Method: We
generate k LDA topics and replicate this process n times resulting in n*k
topics. Then we use K-medioids to cluster the n*k topics to k clusters. The k
clusters now represent the original LDA topics and we present them like normal
LDA topics showing the ten most probable words. For the clusters, we try
multiple stability metrics, out of which we recommend Rank-Biased Overlap,
showing the stability of the topics inside the clusters. Results: We provide an
initial validation where our method is used for 270,000 Mozilla Firefox commit
messages with k=20 and n=20. We show how our topic stability metrics are
related to the contents of the topics. Conclusions: Advances in text mining
enable us to analyze large masses of text in software engineering but
non-deterministic algorithms, such as LDA, may lead to unreplicable
conclusions. Our approach makes LDA stability transparent and is also
complementary rather than alternative to many prior works that focus on LDA
parameter tuning.

我们提出了一种基于重复运行 LDA、聚类和提供主题稳定性度量的方法来增强主题模型的稳定性，并以 Mozilla Firefox 提交信息为例验证了其效果。