Unsupervised pre-trained word embeddings are used effectively for many tasks
in natural language processing to leverage unlabeled textual data. Often these
embeddings are either used as initializations or as fixed word representations
for task-specific classification models. In this work, we extend our
classification model's task loss with an unsupervised auxiliary loss on the
word-embedding level of the model. This is to ensure that the learned word
representations contain both task-specific features, learned from the
supervised loss component, and more general features learned from the
unsupervised loss component. We evaluate our approach on the task of temporal
relation extraction, in particular, narrative containment relation extraction
from clinical records, and show that continued training of the embeddings on
the unsupervised objective together with the task objective gives better
task-specific embeddings, and results in an improvement over the state of the
art on the THYME dataset, using only a general-domain part-of-speech tagger as
linguistic resource.

本文介绍了一种利用无标签文本数据的无监督预训练词向量，通过加入未监督辅助损失（unsupervised auxiliary loss）来扩展分类模型的任务损失，确保学习到的词表示包含了从监督损失组件中学到的 task-specific 特征和从未监督损失组件中学到的更通用的特征，并在临床记录的叙述包含关系抽取任务上对其进行了评估，使用通用类型的词性标注器作为语言资源，在 THYME 数据集上实现了技术上的提高。