Deep neural networks (DNNs) have begun to have a pervasive impact on various
applications of machine learning. However, the problem of finding an optimal
DNN architecture for large applications is challenging. Common approaches go
for deeper and larger DNN architectures but may incur substantial redundancy.
To address these problems, we introduce a network growth algorithm that
complements network pruning to learn both weights and compact DNN architectures
during training. We propose a DNN synthesis tool (NeST) that combines both
methods to automate the generation of compact and accurate DNNs. NeST starts
with a randomly initialized sparse network called the seed architecture. It
iteratively tunes the architecture with gradient-based growth and
magnitude-based pruning of neurons and connections. Our experimental results
show that NeST yields accurate, yet very compact DNNs, with a wide range of
seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we
reduce network parameters by 70.2x (74.3x) and floating-point operations
(FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce
network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively.
NeST's grow-and-prune paradigm delivers significant additional parameter and
FLOPs reduction relative to pruning-only methods.

论文介绍了一种名为 NeST 的 DNN 综合工具，它采用增长和修剪的方式来自动学习权重和紧凑的 DNN 网络架构，以生成准确且很小的 DNN 网络。