To accomplish the punctuation restoration task, most existing approaches
focused on leveraging extra information (e.g., part-of-speech tags) or
addressing the class imbalance problem. Recent works have widely applied the
transformer-based language models and significantly improved their
effectiveness. To the best of our knowledge, an inherent issue has remained
neglected: the attention of individual heads in the transformer will be diluted
or powerless while feeding the long non-punctuation utterances. Since those
previous contexts, not the followings, are comparatively more valuable to the
current position, it's hard to achieve a good balance by independent attention.
In this paper, we propose a novel Feature Fusion framework based on two-type
Attentions (FFA) to alleviate the shortage. It introduces a two-stream
architecture. One module involves interaction between attention heads to
encourage the communication, and another masked attention module captures the
dependent feature representation. Then, it aggregates two feature embeddings to
fuse information and enhances context-awareness. The experiments on the popular
benchmark dataset IWSLT demonstrate that our approach is effective. Without
additional data, it obtains comparable performance to the current
state-of-the-art models.

本文提出了基于两种类型 Attention 的特征融合框架，旨在解决 transformer 模型中存在的长句子注意力不集中问题，实现标点还原任务，并在 IWSLT 数据集上验证其有效性。