The Transformer-based models with the multi-head self-attention mechanism are
widely used in natural language processing, and provide state-of-the-art
results. While the pre-trained language backbones are shown to implicitly
capture certain linguistic knowledge, explicitly incorporating structure-aware
features can bring about further improvement on the downstream tasks. However,
such enhancement often requires additional neural components and increases
training parameter size. In this work, we investigate the attention head
selection and manipulation strategy for feature injection from a network
pruning perspective, and conduct a case study on dialogue summarization. We
first rank attention heads in a Transformer-based summarizer with layer-wise
importance. We then select the underused heads through extensive analysis, and
inject structure-aware features by manipulating the selected heads.
Experimental results show that the importance-based head selection is effective
for feature injection, and dialogue summarization can be improved by
incorporating coreference information via head manipulation.

通过网络修剪的角度，研究了一种特征注入的注意头选择和操作策略，并在对话摘要中进行了案例研究，结果表明通过注意头操作注入指代关系信息可以提高对话摘要的性能。