The Transformer-based models with the multi-head self-attention mechanism are
widely used in natural language processing, and provide state-of-the-art
results. While the pre-trained language backbones are shown to implicitly
capture certain linguistic knowledge, explicitly incorporating structure-aware
features can bring about further improvement on the downstream tasks. However,
such enhancement often requires additional neural components and increases
training parameter size. In this work, we investigate the attention head
selection and manipulation strategy for feature injection from a network
pruning perspective, and conduct a case study on dialogue summarization. We
first rank attention heads in a Transformer-based summarizer with layer-wise
importance. We then select the underused heads through extensive analysis, and
inject structure-aware features by manipulating the selected heads.
Experimental results show that the importance-based head selection is effective
for feature injection, and dialogue summarization can be improved by
incorporating coreference information via head manipulation.

通过网络修剪的角度，研究了一种特征注入的注意头选择和操作策略，并在对话摘要中进行了案例研究，结果表明通过注意头操作注入指代关系信息可以提高对话摘要的性能。

挑选未充分利用的头部：关注网络修剪对于融合对话指代信息的头部选择的视角

Picking the Underused Heads: A Network Pruning Perspective of Attention  Head Selection for Fusing Dialogue Coreference Information

In this paper, we introduce a framework ARBEx, a novel attentive feature
extraction framework driven by Vision Transformer with reliability balancing to
cope against poor class distributions, bias, and uncertainty in the facial
expression learning (FEL) task. We reinforce several data pre-processing and
refinement methods along with a window-based cross-attention ViT to squeeze the
best of the data. We also employ learnable anchor points in the embedding space
with label distributions and multi-head self-attention mechanism to optimize
performance against weak predictions with reliability balancing, which is a
strategy that leverages anchor points, attention scores, and confidence values
to enhance the resilience of label predictions. To ensure correct label
classification and improve the models' discriminative power, we introduce
anchor loss, which encourages large margins between anchor points.
Additionally, the multi-head self-attention mechanism, which is also trainable,
plays an integral role in identifying accurate labels. This approach provides
critical elements for improving the reliability of predictions and has a
substantial positive effect on final prediction capabilities. Our adaptive
model can be integrated with any deep neural network to forestall challenges in
various recognition tasks. Our strategy outperforms current state-of-the-art
methodologies, according to extensive experiments conducted in a variety of
contexts.

本文介绍了一种新的注意力特征提取框架 ARBEx，该框架由 Vision Transformer 驱动，具有可靠性平衡功能，可以应对面部表情学习任务中的类别分布不佳、偏差和不确定性。与基于窗口的交叉注意力 ViT 一起，我们加强了几种数据预处理和细化方法，同时在嵌入空间中引入了学习型锚点、标签分布和多头自我注意机制，以优化性能，减弱标签预测中的弱预测，从而增强标签预测的韧性。同时，引入了锚损失，以鼓励锚点之间的大间隔，提高模型的辨别能力。通过大量实验，在各种情境下，我们的方法优于目前最先进的方法。

ARBEx: 针对健壮面部表情学习的注意力特征提取与可靠性平衡

ARBEx: Attentive Feature Extraction with Reliability Balancing for  Robust Facial Expression Learning

The multi-head self-attention mechanism of the transformer model has been
thoroughly investigated recently. In one vein of study, researchers are
interested in understanding why and how transformers work. In another vein,
researchers propose new attention augmentation methods to make transformers
more accurate, efficient and interpretable. In this paper, we combine these two
lines of research in a human-in-the-loop pipeline to first discover important
task-specific attention patterns. Then those patterns are injected, not only to
smaller models, but also to the original model. The benefits of our pipeline
and discovered patterns are demonstrated in two case studies with extractive
summarization and topic segmentation. After discovering interpretable patterns
in BERT-based models fine-tuned for the two downstream tasks, experiments
indicate that when we inject the patterns into attention heads, the models show
considerable improvements in accuracy and efficiency.

该论文介绍了一种结合人机交互的流程来发现重要的任务特定的注意力模式，然后注入到原始模型和较小的模型来提高模型的准确性与效率，取得了在提取式摘要和主题分割方面显著提高的结果。