Deep neural network ensembles combine the wisdom of multiple deep neural
networks to improve the generalizability and robustness over individual
networks. It has gained increasing popularity to study deep ensemble techniques
in the deep learning community. Some mission-critical applications utilize a
large number of deep neural networks to form deep ensembles to achieve desired
accuracy and resilience, which introduces high time and space costs for
ensemble execution. However, it still remains a critical challenge whether a
small subset of the entire deep ensemble can achieve the same or better
generalizability and how to effectively identify these small deep ensembles for
improving the space and time efficiency of ensemble execution. This paper
presents a novel deep ensemble pruning approach, which can efficiently identify
smaller deep ensembles and provide higher ensemble accuracy than the entire
deep ensemble of a large number of member networks. Our hierarchical ensemble
pruning approach (HQ) leverages three novel ensemble pruning techniques. First,
we show that the focal diversity metrics can accurately capture the
complementary capacity of the member networks of an ensemble, which can guide
ensemble pruning. Second, we design a focal diversity based hierarchical
pruning approach, which will iteratively find high quality deep ensembles with
low cost and high accuracy. Third, we develop a focal diversity consensus
method to integrate multiple focal diversity metrics to refine ensemble pruning
results, where smaller deep ensembles can be effectively identified to offer
high accuracy, high robustness and high efficiency. Evaluated using popular
benchmark datasets, we demonstrate that the proposed hierarchical ensemble
pruning approach can effectively identify high quality deep ensembles with
better generalizability while being more time and space efficient in ensemble
decision making.

该论文提出了一种新的深度集成修剪方法，可以高效地识别较小的深度集成，并提供比大量成员网络的整个深度集成更高的集成准确性。该论文的方法基于三种创新的修剪技术：焦点多样性度量、分层修剪方法和焦点多样性一致性方法，通过这些技术能够在集成决策中更加高效地识别高质量的深度集成。

深度集成的分层剪枝与焦点多样性

Hierarchical Pruning of Deep Ensembles with Focal Diversity

Deep neural network ensembles hold the potential of improving generalization
performance for complex learning tasks. This paper presents formal analysis and
empirical evaluation to show that heterogeneous deep ensembles with high
ensemble diversity can effectively leverage model learning heterogeneity to
boost ensemble robustness. We first show that heterogeneous DNN models trained
for solving the same learning problem, e.g., object detection, can
significantly strengthen the mean average precision (mAP) through our weighted
bounding box ensemble consensus method. Second, we further compose ensembles of
heterogeneous models for solving different learning problems, e.g., object
detection and semantic segmentation, by introducing the connected component
labeling (CCL) based alignment. We show that this two-tier heterogeneity driven
ensemble construction method can compose an ensemble team that promotes high
ensemble diversity and low negative correlation among member models of the
ensemble, strengthening ensemble robustness against both negative examples and
adversarial attacks. Third, we provide a formal analysis of the ensemble
robustness in terms of negative correlation. Extensive experiments validate the
enhanced robustness of heterogeneous ensembles in both benign and adversarial
settings. The source codes are available on GitHub at
this https URL

通过形式分析和实证评估，本文展示了异构深度集成模型可以通过高集成多样性有效地利用模型学习的异质性以提高集成的鲁棒性。

探索用于提升集成强健性的模型学习异质性

Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness

Ensembles of deep neural networks significantly improve generalization
accuracy. However, training neural network ensembles requires a large amount of
computational resources and time. State-of-the-art approaches either train all
networks from scratch leading to prohibitive training cost that allows only
very small ensemble sizes in practice, or generate ensembles by training a
monolithic architecture, which results in lower model diversity and decreased
prediction accuracy. We propose MotherNets to enable higher accuracy and
practical training cost for large and diverse neural network ensembles: A
MotherNet captures the structural similarity across some or all members of a
deep neural network ensemble which allows us to share data movement and
computation costs across these networks. We first train a single or a small set
of MotherNets and, subsequently, we generate the target ensemble networks by
transferring the function from the trained MotherNet(s). Then, we continue to
train these ensemble networks, which now converge drastically faster compared
to training from scratch. MotherNets handle ensembles with diverse
architectures by clustering ensemble networks of similar architecture and
training a separate MotherNet for every cluster. MotherNets also use clustering
to control the accuracy vs. training cost tradeoff. We show that compared to
state-of-the-art approaches such as Snapshot Ensembles, Knowledge Distillation,
and TreeNets, MotherNets provide a new Pareto frontier for the
accuracy-training cost tradeoff. Crucially, training cost and accuracy
improvements continue to scale as we increase the ensemble size (2 to 3 percent
reduced absolute test error rate and up to 35 percent faster training compared
to Snapshot Ensembles). We verify these benefits over numerous neural network
architectures and large data sets.

该研究提出 MotherNets 来解决深度神经网络集成中的训练成本和模型多样性问题，并在减少训练成本和提高模型精度方面取得了新的 Pareto 前沿。