To accomplish punctuation restoration, most existing methods focus on introducing extra information (e.g., part-of-speech) or addressing the class imbalance problem. Recently, large-scale transformer-based pre-trained language models (PLMS) have been utilized widely and obtained remarkable success. However, the PLMS are trained on the large dataset with marks, which may not fit well with the small dataset without marks, causing the convergence to be not ideal. In this study, we propose a Feature Fusion two-stream framework (FF2) to bridge the gap. Specifically, one stream leverages a pre-trained language model to capture the semantic feature, while another auxiliary module captures the feature at hand. We also modify the computation of multi-head attention to encourage communication among heads. Then, two features with different perspectives are aggregated to fuse information and enhance context awareness. Without additional data, the experimental results on the popular benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which verifies that our approach is effective.

提出了一种名为“Feature Fusion two-stream framework (FF2)”的方法，该方法通过将预训练语言模型和另一个辅助模块捆绑起来，同时修改多头注意力的计算方式并将两个不同角度的特征进行融合以增强上下文感知能力，实现了无额外数据下在IWSLT测试集上取得了最新的SOTA表现。

FF2: 用于标点符号恢复的特征融合双流框架