While the depth of modern Convolutional Neural Networks (CNNs) surpasses that
of the pioneering networks with a significant margin, the traditional way of
appending supervision only over the final classifier and progressively
propagating gradient flow upstream remains the training mainstay. Seminal
Deeply-Supervised Networks (DSN) were proposed to alleviate the difficulty of
optimization arising from gradient flow through a long chain. However, it is
still vulnerable to issues including interference to the hierarchical
representation generation process and inconsistent optimization objectives, as
illustrated theoretically and empirically in this paper. Complementary to
previous training strategies, we propose Dynamic Hierarchical Mimicking, a
generic feature learning mechanism, to advance CNN training with enhanced
generalization ability. Partially inspired by DSN, we fork delicately designed
side branches from the intermediate layers of a given neural network. Each
branch can emerge from certain locations of the main branch dynamically, which
not only retains representation rooted in the backbone network but also
generates more diverse representations along its own pathway. We go one step
further to promote multi-level interactions among different branches through an
optimization formula with probabilistic prediction matching losses, thus
guaranteeing a more robust optimization process and better representation
ability. Experiments on both category and instance recognition tasks
demonstrate the substantial improvements of our proposed method over its
corresponding counterparts using diverse state-of-the-art CNN architectures.
Code and models are publicly available at this https URL

介绍了一种名为动态分层模仿的泛化能力增强的卷积神经网络训练方法，通过设计与主干网络分支相连的支路，实现路径多样性和更多样化的特征表示，并应用概率预测匹配损失优化公式促进不同分支的多级交互，提高网络训练质量和特征表示能力。