Compressed Neural Networks have the potential to enable deep learning across
new applications and smaller computational environments. However, understanding
the range of learning tasks in which such models can succeed is not well
studied. In this work, we apply sparse and binary-weighted Transformers to
multivariate time series problems, showing that the lightweight models achieve
accuracy comparable to that of dense floating-point Transformers of the same
structure. Our model achieves favorable results across three time series
learning tasks: classification, anomaly detection, and single-step forecasting.
Additionally, to reduce the computational complexity of the attention
mechanism, we apply two modifications, which show little to no decline in model
performance: 1) in the classification task, we apply a fixed mask to the query,
key, and value activations, and 2) for forecasting and anomaly detection, which
rely on predicting outputs at a single point in time, we propose an attention
mask to allow computation only at the current time step. Together, each
compression technique and attention modification substantially reduces the
number of non-zero operations necessary in the Transformer. We measure the
computational savings of our approach over a range of metrics including
parameter count, bit size, and floating point operation (FLOPs) count, showing
up to a 53x reduction in storage size and up to 10.5x reduction in FLOPs.

通过应用稀疏和二元权重变换器在多变量时间序列问题上，我们表明轻量级模型能够获得与相同结构的稠密浮点变换器相当的准确性。我们的模型在分类、异常检测和单步预测三个时间序列学习任务上取得了良好的结果，并应用了两种修改以减少注意力机制的计算复杂度，从而大大减少变换器中非零操作的数量。我们在参数数量、存储大小和浮点运算量（FLOPs）等多个度量标准上测量了我们的方法的计算节约，并展示了高达 53 倍的存储大小减少和高达 10.5 倍的 FLOPs 减少。