Machine learning algorithms often contain many hyperparameters whose values affect the predictive performance of the induced models in intricate ways. Due to the high number of possibilities for these hyperparameter configurations, and their complex interactions, it is common to use optimization techniques to find settings that lead to high predictive accuracy. However, we lack insight into how to efficiently explore this vast space of configurations: which are the best optimization techniques, how should we use them, and how significant is their effect on predictive or runtime performance? This paper provides a comprehensive approach for investigating the effects of hyperparameter tuning on three Decision Tree induction algorithms, CART, C4.5 and CTree. These algorithms were selected because they are based on similar principles, have presented a high predictive performance in several previous works and induce interpretable classification models. Additionally, they contain many interacting hyperparameters to be adjusted. Experiments were carried out with different tuning strategies to induce models and evaluate the relevance of hyperparameters using 94 classification datasets from OpenML. Experimental results indicate that hyperparameter tuning provides statistically significant improvements for C4.5 and CTree in only one-third of the datasets, and in most of the datasets for CART. Different tree algorithms may present different tuning scenarios, but in general, the tuning techniques required relatively few iterations to find accurate solutions. Furthermore, the best technique for all the algorithms was the Irace. Finally, we find that tuning a specific small subset of hyperparameters contributes most of the achievable optimal predictive performance.

本文提供了一种全面的方法，用于研究超参数调整对CART，C4.5和CTree三种决策树归纳算法的影响。实验结果表明，超参数调整仅在三分之一的数据集中为C4.5和CTree提供了显著的改进，在大多数数据集中却为CART提供了显著的改进。不同的树算法可能呈现不同的调优场景，但总体而言，调优技术仅需要很少的迭代就能找到准确的解决方案。此外，所有算法的最佳技术都是Irace。最后，我们发现调整特定的一小部分超参数对于可达到的最佳预测性能做出了最大的贡献。

决策树超参数调整的实证研究