Deep pre-trained Transformer models have achieved state-of-the-art results
over a variety of natural language processing (NLP) tasks. By learning rich
language knowledge with millions of parameters, these models are usually
overparameterized and significantly increase the computational overhead in
applications. It is intuitive to address this issue by model compression. In
this work, we propose a method, called Single-Shot Meta-Pruning, to compress
deep pre-trained Transformers before fine-tuning. Specifically, we focus on
pruning unnecessary attention heads adaptively for different downstream tasks.
To measure the informativeness of attention heads, we train our Single-Shot
Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the
distribution of text representations after pruning. Compared with existing
compression methods for pre-trained models, our method can reduce the overhead
of both fine-tuning and inference. Experimental results show that our pruner
can selectively prune 50% of attention heads with little impact on the
performance on downstream tasks and even provide better text representations.
The source code will be released in the future.

本文提出了一种称为 Single-Shot Meta-Pruning 的方法，该方法致力于压缩深度预训练的 Transformer 模型，并集中于可以自适应地为不同的下游任务剪枝不必要的注意力头。与现有的预训练模型压缩方法相比，我们的方法可以降低精调和推理的开销，并且可以选择性地剪枝 50％的注意力头，对下游任务的性能几乎没有影响，甚至提供更好的文本表示。

单次元元剪枝：减少注意力头部无关部分

Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

Answer Set Programming (ASP) is a powerful logic-based programming language,
which is enjoying increasing interest within the scientific community and (very
recently) in industry. The evaluation of ASP programs is traditionally carried
out in two steps. At the first step an input program P undergoes the so-called
instantiation (or grounding) process, which produces a program P' semantically
equivalent to P, but not containing any variable; in turn, P' is evaluated by
using a backtracking search algorithm in the second step. It is well-known that
instantiation is important for the efficiency of the whole evaluation, might
become a bottleneck in common situations, is crucial in several realworld
applications, and is particularly relevant when huge input data has to be dealt
with. At the time of this writing, the available instantiator modules are not
able to exploit satisfactorily the latest hardware, featuring
multi-core/multi-processor SMP (Symmetric MultiProcessing) technologies. This
paper presents some parallel instantiation techniques, including load-balancing
and granularity control heuristics, which allow for the effective exploitation
of the processing power offered by modern SMP machines. This is confirmed by an
extensive experimental analysis herein reported.
To appear in Theory and Practice of Logic Programming (TPLP).
KEYWORDS: Answer Set Programming, Instantiation, Parallelism, Heuristics

本研究介绍了一些并行的实例化技术，包括负载平衡和粒度控制启发式方法，可有效利用现代 SMP 机器所提供的处理能力。