Due to complex interactions among various deep neural network (DNN)
optimization techniques, modern DNNs can have weights and activations that are
dense or sparse with diverse sparsity degrees. To offer a good trade-off
between accuracy and hardware performance, an ideal DNN accelerator should have
high flexibility to efficiently translate DNN sparsity into reductions in
energy and/or latency without incurring significant complexity overhead.
This paper introduces hierarchical structured sparsity (HSS), with the key
insight that we can systematically represent diverse sparsity degrees by having
them hierarchically composed from multiple simple sparsity patterns. As a
result, HSS simplifies the underlying hardware since it only needs to support
simple sparsity patterns; this significantly reduces the sparsity acceleration
overhead, which improves efficiency. Motivated by such opportunities, we
propose a simultaneously efficient and flexible accelerator, named HighLight,
to accelerate DNNs that have diverse sparsity degrees (including dense). Due to
the flexibility of HSS, different HSS patterns can be introduced to DNNs to
meet different applications' accuracy requirements. Compared to existing works,
HighLight achieves a geomean of up to 6.4x better energy-delay product (EDP)
across workloads with diverse sparsity degrees, and always sits on the
EDP-accuracy Pareto frontier for representative DNNs.

以多层复杂结构表示不同稀疏度，提出 DNN 加速器 HighLight，能有效地将 DNN 稀疏化转换为降低能耗和延迟的技术，并达到高灵活性和性能，改善 DNN 应用的精度和能效。

高亮：基于分层结构稀疏性的高效灵活的 DNN 加速

HighLight: Efficient and Flexible DNN Acceleration with Hierarchical  Structured Sparsity

Across applications spanning supervised classification and sequential
control, deep learning has been reported to find "shortcut" solutions that fail
catastrophically under minor changes in the data distribution. In this paper,
we show empirically that DNNs can be coaxed to avoid poor shortcuts by
providing an additional "priming" feature computed from key input features,
usually a coarse output estimate. Priming relies on approximate domain
knowledge of these task-relevant key input features, which is often easy to
obtain in practical settings. For example, one might prioritize recent frames
over past frames in a video input for visual imitation learning, or salient
foreground over background pixels for image classification. On NICO image
classification, MuJoCo continuous control, and CARLA autonomous driving, our
priming strategy works significantly better than several popular
state-of-the-art approaches for feature selection and data augmentation. We
connect these empirical findings to recent theoretical results on DNN
optimization, and argue theoretically that priming distracts the optimizer away
from poor shortcuts by creating better, simpler shortcuts.

本文通过实证分析表明新加基于先验知识的 priming 特征（例如通过视觉学习中的最近一帧或者图像分类中的醒目前景进行分类）可以显著提高深度学习对图像分类、持续控制和自主驾驶等应用的效果，从而较好地避免训练时出现的 “短路” 现象。理论上，这是因为 priming 特征引导了模型优化的方向，使其避免陷入一些不可靠的简单捷径。

火中取栗：通过引导避免 DNN 捷径

Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming

We present novel empirical observations regarding how stochastic gradient
descent (SGD) navigates the loss landscape of over-parametrized deep neural
networks (DNNs). These observations expose the qualitatively different roles of
learning rate and batch-size in DNN optimization and generalization.
Specifically we study the DNN loss surface along the trajectory of SGD by
interpolating the loss surface between parameters from consecutive
\textit{iterations} and tracking various metrics during training. We find that
the loss interpolation between parameters before and after each training
iteration's update is roughly convex with a minimum (\textit{valley floor}) in
between for most of the training. Based on this and other metrics, we deduce
that for most of the training update steps, SGD moves in valley like regions of
the loss surface by jumping from one valley wall to another at a height above
the valley floor. This 'bouncing between walls at a height' mechanism helps SGD
traverse larger distance for small batch sizes and large learning rates which
we find play qualitatively different roles in the dynamics. While a large
learning rate maintains a large height from the valley floor, a small batch
size injects noise facilitating exploration. We find this mechanism is crucial
for generalization because the valley floor has barriers and this exploration
above the valley floor allows SGD to quickly travel far away from the
initialization point (without being affected by barriers) and find flatter
regions, corresponding to better generalization.

我们研究了随机梯度下降在超参数化深度神经网络的损失面上如何导航，发现学习率和批量大小在深度神经网络优化和泛化中具有不同的作用，并且 ' 在高度上在墙之间弹跳 ' 机制对于泛化至关重要。