We present STAT: a simple algorithm to prune transformer models without any
fine-tuning. STAT eliminates both attention heads and neurons from the network,
while preserving accuracy by calculating a correction to the weights of the
next layer. Each layer block in the network is compressed using a series of
principled matrix factorizations that preserve the network structure. Our
entire algorithm takes minutes to compress BERT, and less than three hours to
compress models with 7B parameters using a single GPU. Using only several
hundred data examples, STAT preserves the output of the network and improves
upon existing gradient-free pruning methods. It is even competitive with
methods that include significant fine-tuning. We demonstrate our method on both
encoder and decoder architectures, including BERT, DistilBERT, and Llama-2
using benchmarks such as GLUE, Squad, WikiText2.

我们提出了 STAT 算法，一种简单的算法用于修剪 transformer 模型，而无需进行任何微调。STAT 通过计算下一层权重的修正来压缩网络中的注意力头和神经元，同时保持精度，并使用一系列有原则的矩阵分解方法来压缩网络结构。该算法仅需要几分钟来压缩 BERT，并且在单个 GPU 上使用不到三个小时来压缩具有 7B 参数的模型。STAT 仅使用几百个数据示例即可保留网络输出并改进现有的无梯度修剪方法，甚至能与包括重要微调的方法竞争。我们在编码器和解码器体系结构上展示了该方法，包括 BERT，DistilBERT 和 Llama-2，并使用 GLUE，Squad 和 WikiText2 等基准测试。

STAT：训练后的 Transformer 收缩

STAT: Shrinking Transformers After Training

Weakly-supervised temporal action localization (WTAL) aims to recognize and
localize action instances with only video-level labels. Despite the significant
progress, existing methods suffer from severe performance degradation when
transferring to different distributions and thus may hardly adapt to real-world
scenarios . To address this problem, we propose the Generalizable Temporal
Action Localization task (GTAL), which focuses on improving the
generalizability of action localization methods. We observed that the
performance decline can be primarily attributed to the lack of generalizability
to different action scales. To address this problem, we propose STAT
(Self-supervised Temporal Adaptive Teacher), which leverages a teacher-student
structure for iterative refinement. Our STAT features a refinement module and
an alignment module. The former iteratively refines the model's output by
leveraging contextual information and helps adapt to the target scale. The
latter improves the refinement process by promoting a consensus between student
and teacher models. We conduct extensive experiments on three datasets,
THUMOS14, ActivityNet1.2, and HACS, and the results show that our method
significantly improves the Baseline methods under the cross-distribution
evaluation setting, even approaching the same-distribution evaluation
performance.

提出了一种弱监督时序动作定位算法 STAT，通过引入自监督时序自适应教师模块和对齐模块，在不同数据分布下显著提高了动作定位方法的泛化能力。