3D convolutional neural networks have revealed superior performance in
processing volumetric data such as video and medical imaging. However, the
competitive performance by leveraging 3D networks results in huge computational
costs, which are far beyond that of 2D networks. In this paper, we propose a
novel Hilbert curve-based cross-dimensionality distillation approach that
facilitates the knowledge of 3D networks to improve the performance of 2D
networks. The proposed Hilbert Distillation (HD) method preserves the
structural information via the Hilbert curve, which maps high-dimensional (>=2)
representations to one-dimensional continuous space-filling curves. Since the
distilled 2D networks are supervised by the curves converted from dimensionally
heterogeneous 3D features, the 2D networks are given an informative view in
terms of learning structural information embedded in well-trained
high-dimensional representations. We further propose a Variable-length Hilbert
Distillation (VHD) method to dynamically shorten the walking stride of the
Hilbert curve in activation feature areas and lengthen the stride in context
feature areas, forcing the 2D networks to pay more attention to learning from
activation features. The proposed algorithm outperforms the current
state-of-the-art distillation techniques adapted to cross-dimensionality
distillation on two classification tasks. Moreover, the distilled 2D networks
by the proposed method achieve competitive performance with the original 3D
networks, indicating the lightweight distilled 2D networks could potentially be
the substitution of cumbersome 3D networks in the real-world scenario.

本文提出了一种全新的基于希尔伯特曲线的跨维度蒸馏方法 (HD)，通过将 3D 特征转换为一维连续的空间填充曲线，将 3D 网络的知识迁移到提高 2D 网络性能，进一步提出了一种动态调整步幅的变长度希尔伯特蒸馏方法 (VHD)，可以更好地学习激活特征，相比现有的跨维度蒸馏技术，实验表明本文提出的算法在两个分类任务上性能更优，而且所蒸馏的 2D 网络具有与 3D 网络相媲美的性能水平，证明轻量级的蒸馏 2D 网络可以成为麻烦的 3D 网络的替代品之一