Previous studies demonstrate that word embeddings and part-of-speech (POS)
tags are helpful for punctuation restoration tasks. However, two drawbacks
still exist. One is that word embeddings are pre-trained by unidirectional
language modeling objectives. Thus the word embeddings only contain
left-to-right context information. The other is that POS tags are provided by
an external POS tagger. So computation cost will be increased and incorrect
predicted tags may affect the performance of restoring punctuation marks during
decoding. This paper proposes adversarial transfer learning to address these
problems. A pre-trained bidirectional encoder representations from transformers
(BERT) model is used to initialize a punctuation model. Thus the transferred
model parameters carry both left-to-right and right-to-left representations.
Furthermore, adversarial multi-task learning is introduced to learn task
invariant knowledge for punctuation prediction. We use an extra POS tagging
task to help the training of the punctuation predicting task. Adversarial
training is utilized to prevent the shared parameters from containing task
specific information. We only use the punctuation predicting task to restore
marks during decoding stage. Therefore, it will not need extra computation and
not introduce incorrect tags from the POS tagger. Experiments are conducted on
IWSLT2011 datasets. The results demonstrate that the punctuation predicting
models obtain further performance improvement with task invariant knowledge
from the POS tagging task. Our best model outperforms the previous
state-of-the-art model trained only with lexical features by up to 9.2%
absolute overall F_1-score on test set.

本研究旨在解决文本标点还原中词向量只包含从左到右上下文信息和外部词性标注器的不足。为此，引入敌对式迁移学习和多任务联合学习，将预训练双向编码器表示模型用于初始化标点模型，利用多个任务的学习提高模型效果，在 IWSLT2011 数据集上进行实验，结果表明本文提出的模型相较于以往基于词法特征的模型，在测试集上的 F1-score 表现提高了 9.2%。