Recurrent Neural Networks (RNNs) are used in state-of-the-art models in
domains such as speech recognition, machine translation, and language
modelling. Sparsity is a technique to reduce compute and memory requirements of
deep learning models. Sparse RNNs are easier to deploy on devices and high-end
server processors. Even though sparse operations need less compute and memory
relative to their dense counterparts, the speed-up observed by using sparse
operations is less than expected on different hardware platforms. In order to
address this issue, we investigate two different approaches to induce block
sparsity in RNNs: pruning blocks of weights in a layer and using group lasso
regularization to create blocks of weights with zeros. Using these techniques,
we demonstrate that we can create block-sparse RNNs with sparsity ranging from
80% to 90% with small loss in accuracy. This allows us to reduce the model size
by roughly 10x. Additionally, we can prune a larger dense network to recover
this loss in accuracy while maintaining high block sparsity and reducing the
overall parameter count. Our technique works with a variety of block sizes up
to 32x32. Block-sparse RNNs eliminate overheads related to data storage and
irregular memory accesses while increasing hardware efficiency compared to
unstructured sparsity.

研究使用 “修剪” 和 “组稀疏化正则化” 这两种方法来诱导 RNN 网络中的块稀疏性，形成 80% 到 90% 不等的稀疏度；该技术能够消除与数据存储和不规则内存访问相关的额外开销，同时提高硬件效率和降低参数总数。

基块稀疏循环神经网络

Block-Sparse Recurrent Neural Networks

Model compression is significant for the wide adoption of Recurrent Neural
Networks (RNNs) in both user devices possessing limited resources and business
clusters requiring quick responses to large-scale service requests. This work
aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the
sizes of basic structures within LSTM units, including input updates, gates,
hidden states, cell states and outputs. Independently reducing the sizes of
basic structures can result in inconsistent dimensions among them, and
consequently, end up with invalid LSTM units. To overcome the problem, we
propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS
will simultaneously decrease the sizes of all basic structures by one and
thereby always maintain the dimension consistency. By learning ISS within LSTM
units, the obtained LSTMs remain regular while having much smaller basic
structures. Based on group Lasso regularization, our method achieves 10.59x
speedup without losing any perplexity of a language modeling of Penn TreeBank
dataset. It is also successfully evaluated through a compact model with only
2.69M weights for machine Question Answering of SQuAD dataset. Our approach is
successfully extended to non- LSTM RNNs, like Recurrent Highway Networks
(RHNs). Our source code is publicly available at
this https URL

该论文提出了一种内在稀疏结构方法，通过独立地减小 LSTM 单元内基本结构的尺寸并保持尺寸一致性来实现结构上稀疏的 LSTM，这种方法可以显著提高模型的速度而不会损失模型性能，并成功地被应用于不同的 RNN 架构。

学习长短期记忆内在稀疏结构

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Future projection of climate is typically obtained by combining outputs from
multiple Earth System Models (ESMs) for several climate variables such as
temperature and precipitation. While IPCC has traditionally used a simple model
output average, recent work has illustrated potential advantages of using a
multitask learning (MTL) framework for projections of individual climate
variables. In this paper we introduce a framework for hierarchical multitask
learning (HMTL) with two levels of tasks such that each super-task, i.e., task
at the top level, is itself a multitask learning problem over sub-tasks. For
climate projections, each super-task focuses on projections of specific climate
variables spatially using an MTL formulation. For the proposed HMTL approach, a
group lasso regularization is added to couple parameters across the
super-tasks, which in the climate context helps exploit relationships among the
behavior of different climate variables at a given spatial location. We show
that some recent works on MTL based on learning task dependency structures can
be viewed as special cases of HMTL. Experiments on synthetic and real climate
data show that HMTL produces better results than decoupled MTL methods applied
separately on the super-tasks and HMTL significantly outperforms baselines for
climate projection.

提出了一种基于层次多任务学习框架（HMTL）的气候投影方法，该方法使用多任务学习公式实现特定气候变量的空间投影，并添加了组套索正则化以便在给定的空间位置上利用不同气候变量之间的关系。结果显示相比于其他方法，HMTL 方法可以更有效地实现气候变量的投影。