Deep neural networks have achieved remarkable success in a wide range of
practical problems. However, due to the inherent large parameter space, deep
models are notoriously prone to overfitting and difficult to be deployed in
portable devices with limited memory. In this paper, we propose an iterative
hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs).
An SDNN has much fewer parameters yet can achieve competitive or even better
performance than its full CNN counterpart. More concretely, the IHT approach
trains an SDNN through following two alternative phases: (I) perform hard
thresholding to drop connections with small activations and fine-tune the other
significant filters; (II)~re-activate the frozen connections and train the
entire network to improve its overall discriminative capability. We verify the
superiority of SDNNs in terms of efficiency and classification performance on
four benchmark object recognition datasets, including CIFAR-10, CIFAR-100,
MNIST and ImageNet. Experimental results clearly demonstrate that IHT can be
applied for training SDNN based on various CNN architectures such as NIN and
AlexNet.

本研究提出了一种迭代硬阈值（IHT）方法用于训练 “瘦” 深度神经网络（SDNN），该方法可以通过两个阶段的交替训练操作，即丢弃小激活连接并微调其他重要权重，然后重新激活被丢弃的连接以提高模型疏密性。该方法的优越性通过使用 CIFAR-10，CIFAR-100，MNIST 和 ImageNet 四种基准数据集的实验结果得到了验证。