Compressed Neural Networks have the potential to enable deep learning across
new applications and smaller computational environments. However, understanding
the range of learning tasks in which such models can succeed is not well
studied. In this work, we apply sparse and binary-weighted Transformers to
multivariate time series problems, showing that the lightweight models achieve
accuracy comparable to that of dense floating-point Transformers of the same
structure. Our model achieves favorable results across three time series
learning tasks: classification, anomaly detection, and single-step forecasting.
Additionally, to reduce the computational complexity of the attention
mechanism, we apply two modifications, which show little to no decline in model
performance: 1) in the classification task, we apply a fixed mask to the query,
key, and value activations, and 2) for forecasting and anomaly detection, which
rely on predicting outputs at a single point in time, we propose an attention
mask to allow computation only at the current time step. Together, each
compression technique and attention modification substantially reduces the
number of non-zero operations necessary in the Transformer. We measure the
computational savings of our approach over a range of metrics including
parameter count, bit size, and floating point operation (FLOPs) count, showing
up to a 53x reduction in storage size and up to 10.5x reduction in FLOPs.

通过应用稀疏和二元权重变换器在多变量时间序列问题上，我们表明轻量级模型能够获得与相同结构的稠密浮点变换器相当的准确性。我们的模型在分类、异常检测和单步预测三个时间序列学习任务上取得了良好的结果，并应用了两种修改以减少注意力机制的计算复杂度，从而大大减少变换器中非零操作的数量。我们在参数数量、存储大小和浮点运算量（FLOPs）等多个度量标准上测量了我们的方法的计算节约，并展示了高达 53 倍的存储大小减少和高达 10.5 倍的 FLOPs 减少。

稀疏二进制变换器用于多元时间序列建模

Sparse Binary Transformers for Multivariate Time Series Modeling

The rapidly growing parameter volume of deep neural networks (DNNs) hinders
the artificial intelligence applications on resource constrained devices, such
as mobile and wearable devices. Neural network pruning, as one of the
mainstream model compression techniques, is under extensive study to reduce the
number of parameters and computations. In contrast to irregular pruning that
incurs high index storage and decoding overhead, structured pruning techniques
have been proposed as the promising solutions. However, prior studies on
structured pruning tackle the problem mainly from the perspective of
facilitating hardware implementation, without analyzing the characteristics of
sparse neural networks. The neglect on the study of sparse neural networks
causes inefficient trade-off between regularity and pruning ratio.
Consequently, the potential of structurally pruning neural networks is not
sufficiently mined.
In this work, we examine the structural characteristics of the irregularly
pruned weight matrices, such as the diverse redundancy of different rows, the
sensitivity of different rows to pruning, and the positional characteristics of
retained weights. By leveraging the gained insights as a guidance, we first
propose the novel block-max weight masking (BMWM) method, which can effectively
retain the salient weights while imposing high regularity to the weight matrix.
As a further optimization, we propose a density-adaptive regular-block (DARB)
pruning that outperforms prior structured pruning work with high pruning ratio
and decoding efficiency. Our experimental results show that DARB can achieve
13$\times$ to 25$\times$ pruning ratio, which are 2.8$\times$ to 4.3$\times$
improvements than the state-of-the-art counterparts on multiple neural network
models and tasks. Moreover, DARB can achieve 14.3$\times$ decoding efficiency
than block pruning with higher pruning ratio.

本研究基于先前研究结构性剪枝技术，从稀疏神经网络的角度分析剪枝问题，提出 Block-Max Weight Masking (BMWM) 和 Density-Adaptive Regular-Block (DARB) 两种新方法，比现实有人工智能领域的现有结构化剪枝技术提高了 13-25 倍和 14.3 倍的剪枝和解码效率。