Transformer models have been widely adopted in various domains over the last
years, and especially large language models have advanced the field of AI
significantly. Due to their size, the capability of these networks has
increased tremendously, but this has come at the cost of a significant increase
in necessary compute. Quantization is one of the most effective ways to reduce
the computational time and memory consumption of neural networks. Many studies
have shown, however, that modern transformer models tend to learn strong
outliers in their activations, making them difficult to quantize. To retain
acceptable performance, the existence of these outliers requires activations to
be in higher bitwidth or the use of different numeric formats, extra
fine-tuning, or other workarounds. We show that strong outliers are related to
very specific behavior of attention heads that try to learn a "no-op" or just a
partial update of the residual. To achieve the exact zeros needed in the
attention matrix for a no-update, the input to the softmax is pushed to be
larger and larger during training, causing outliers in other parts of the
network. Based on these observations, we propose two simple (independent)
modifications to the attention mechanism - clipped softmax and gated attention.
We empirically show that models pre-trained using our methods learn
significantly smaller outliers while maintaining and sometimes even improving
the floating-point task performance. This enables us to quantize transformers
to full INT8 quantization of the activations without any additional effort. We
demonstrate the effectiveness of our methods on both language models (BERT,
OPT) and vision transformers.

研究人员提出了两种 Attention 机制的变化：Clipped softmax 和 Gated attention，它们可以帮助模型学习更小的异常值。利用这些变化，模型可以进行完整的 INT8 激活量化而无需额外的努力。

可量化的 Transformer 模型：通过助于自注意力机制头部不做多余操作消除异常值

Quantizable Transformers: Removing Outliers by Helping Attention Heads  Do Nothing

Scene text removal (STR), a task of erasing text from natural scene images,
has recently attracted attention as an important component of editing text or
concealing private information such as ID, telephone, and license plate
numbers. While there are a variety of different methods for STR actively being
researched, it is difficult to evaluate superiority because previously proposed
methods do not use the same standardized training/evaluation dataset. We use
the same standardized training/testing dataset to evaluate the performance of
several previous methods after standardized re-implementation. We also
introduce a simple yet extremely effective Gated Attention (GA) and
Region-of-Interest Generation (RoIG) methodology in this paper. GA uses
attention to focus on the text stroke as well as the textures and colors of the
surrounding regions to remove text from the input image much more precisely.
RoIG is applied to focus on only the region with text instead of the entire
image to train the model more efficiently. Experimental results on the
benchmark dataset show that our method significantly outperforms existing
state-of-the-art methods in almost all metrics with remarkably higher-quality
results. Furthermore, because our model does not generate a text stroke mask
explicitly, there is no need for additional refinement steps or sub-models,
making our model extremely fast with fewer parameters. The dataset and code are
available at this this https URL

使用相同的标准化训练 / 评估数据集对多种先前的方法进行评估，提出了一种简单而极其有效的门控注意力（GA）和感兴趣区域生成（RoIG）方法，证明了这种方法在几乎所有指标上都明显优于现有的最先进方法，速度更快，参数更少。

利用门控注意力和感兴趣区域生成的惊人简单的场景文本去除方法：全面突出模型分析

The Surprisingly Straightforward Scene Text Removal Method With Gated  Attention and Region of Interest Generation: A Comprehensive Prominent Model  Analysis