Neural networks (NNs) and decision trees (DTs) are both popular models of
machine learning, yet coming with mutually exclusive advantages and
limitations. To bring the best of the two worlds, a variety of approaches are
proposed to integrate NNs and DTs explicitly or implicitly. In this survey,
these approaches are organized in a school which we term as neural trees (NTs).
This survey aims to present a comprehensive review of NTs and attempts to
identify how they enhance the model interpretability. We first propose a
thorough taxonomy of NTs that expresses the gradual integration and
co-evolution of NNs and DTs. Afterward, we analyze NTs in terms of their
interpretability and performance, and suggest possible solutions to the
remaining challenges. Finally, this survey concludes with a discussion about
other considerations like conditional computation and promising directions
towards this field. A list of papers reviewed in this survey, along with their
corresponding codes, is available at:
this https URL

本文综述了神经树 (NTs) 方法，以实现神经网络（NNs）和决策树（DTs）模型的有机融合，探讨了如何提高模型的可解释性，并提出了解决方案。

神经树综述

A Survey of Neural Trees

Understanding the source of the superior generalization ability of NNs
remains one of the most important problems in ML research. There have been a
series of theoretical works trying to derive non-vacuous bounds for NNs.
Recently, the compression of information stored in weights (IIW) is proved to
play a key role in NNs generalization based on the PAC-Bayes theorem. However,
no solution of IIW has ever been provided, which builds a barrier for further
investigation of the IIW's property and its potential in practical deep
learning. In this paper, we propose an algorithm for the efficient
approximation of IIW. Then, we build an IIW-based information bottleneck on the
trade-off between accuracy and information complexity of NNs, namely PIB. From
PIB, we can empirically identify the fitting to compressing phase transition
during NNs' training and the concrete connection between the IIW compression
and the generalization. Besides, we verify that IIW is able to explain NNs in
broad cases, e.g., varying batch sizes, over-parameterization, and noisy
labels. Moreover, we propose an MCMC-based algorithm to sample from the optimal
weight posterior characterized by PIB, which fulfills the potential of IIW in
enhancing NNs in practice.

提出了一种有效的算法来近似存储在权重中的信息（IIW）并基于此构建了一种基于准确性和信息复杂度之间的权衡的 IIW 信息瓶颈（PIB），从中可以在 NN 培训期间经验性地确定压缩相变的适应性和 IIW 压缩与泛化之间的具体联系。此外，还提出了一种基于 MCMC 的算法，用于从 PIB 表征的最优权重后验中进行采样，从而实现了 IIW 在增强 NN 方面的潜力。