Deep neural networks (DNNs) have begun to have a pervasive impact on various
applications of machine learning. However, the problem of finding an optimal
DNN architecture for large applications is challenging. Common approaches go
for deeper and larger DNN architectures but may incur substantial redundancy.
To address these problems, we introduce a network growth algorithm that
complements network pruning to learn both weights and compact DNN architectures
during training. We propose a DNN synthesis tool (NeST) that combines both
methods to automate the generation of compact and accurate DNNs. NeST starts
with a randomly initialized sparse network called the seed architecture. It
iteratively tunes the architecture with gradient-based growth and
magnitude-based pruning of neurons and connections. Our experimental results
show that NeST yields accurate, yet very compact DNNs, with a wide range of
seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we
reduce network parameters by 70.2x (74.3x) and floating-point operations
(FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce
network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively.
NeST's grow-and-prune paradigm delivers significant additional parameter and
FLOPs reduction relative to pruning-only methods.

论文介绍了一种名为 NeST 的 DNN 综合工具，它采用增长和修剪的方式来自动学习权重和紧凑的 DNN 网络架构，以生成准确且很小的 DNN 网络。

基于生长 - 修剪范式的神经网络合成工具 NeST

NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

When approaching a novel visual recognition problem in a specialized image
domain, a common strategy is to start with a pre-trained deep neural network
and fine-tune it to the specialized domain. If the target domain covers a
smaller visual space than the source domain used for pre-training (e.g.
ImageNet), the fine-tuned network is likely to be over-parameterized. However,
applying network pruning as a post-processing step to reduce the memory
requirements has drawbacks: fine-tuning and pruning are performed
independently; pruning parameters are set once and cannot adapt over time; and
the highly parameterized nature of state-of-the-art pruning methods make it
prohibitive to manually search the pruning parameter space for deep networks,
leading to coarse approximations. We propose a principled method for jointly
fine-tuning and compressing a pre-trained convolutional network that overcomes
these limitations. Experiments on two specialized image domains (remote sensing
images and describable textures) demonstrate the validity of the proposed
approach.

本研究提出了一种基于卷积神经网络的精调和压缩方法，旨在同时克服细调和修剪在参数空间上的问题，实验结果显示该方法在两个特殊图像领域有效。

Fine-Pruning：卷积网络的联合微调和压缩与贝叶斯优化

Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional  Network with Bayesian Optimization

Real time application of deep learning algorithms is often hindered by high
computational complexity and frequent memory accesses. Network pruning is a
promising technique to solve this problem. However, pruning usually results in
irregular network connections that not only demand extra representation efforts
but also do not fit well on parallel computation. We introduce structured
sparsity at various scales for convolutional neural networks, which are channel
wise, kernel wise and intra kernel strided sparsity. This structured sparsity
is very advantageous for direct computational resource savings on embedded
computers, parallel computing environments and hardware based systems. To
decide the importance of network connections and paths, the proposed method
uses a particle filtering approach. The importance weight of each particle is
assigned by computing the misclassification rate with corresponding
connectivity pattern. The pruned network is re-trained to compensate for the
losses due to pruning. While implementing convolutions as matrix products, we
particularly show that intra kernel strided sparsity with a simple constraint
can significantly reduce the size of kernel and feature map matrices. The
pruned network is finally fixed point optimized with reduced word length
precision. This results in significant reduction in the total storage size
providing advantages for on-chip memory based implementations of deep neural
networks.

使用结构化稀疏技术、粒子滤波方法和固定点优化技术对深度神经网络进行了精简，优化后可在嵌入式计算机和硬件系统上提供更加高效和优化的实现。