Recent advances, such as GPT and BERT, have shown success in incorporating a
pre-trained transformer language model and fine-tuning operation to improve
downstream NLP systems. However, this framework still has some fundamental
problems in effectively incorporating supervised knowledge from other related
tasks. In this study, we investigate a transferable BERT (TransBERT) training
framework, which can transfer not only general language knowledge from
large-scale unlabeled data but also specific kinds of knowledge from various
semantically related supervised tasks, for a target task. Particularly, we
propose utilizing three kinds of transfer tasks, including natural language
inference, sentiment classification, and next action prediction, to further
train BERT based on a pre-trained model. This enables the model to get a better
initialization for the target task. We take story ending prediction as the
target task to conduct experiments. The final result, an accuracy of 91.8%,
dramatically outperforms previous state-of-the-art baseline methods. Several
comparative experiments give some helpful suggestions on how to select transfer
tasks. Error analysis shows what are the strength and weakness of BERT-based
models for story ending prediction.

本研究旨在研究一种可转移的 BERT 训练框架，该框架不仅可以从大规模未标记数据中转移一般语言知识，还可以从各种语义相关的监督任务中转移特定类型的知识以用于目标任务。在本实验中，提出了利用三种转移任务，包括自然语言推理、情感分类和下一动作预测，以进一步训练预训练模型的思想，并将故事结尾预测作为目标任务进行实验。最终结果，准确率达到了 91.8％，明显优于先前最先进的基线方法。同时，进行了一些比较实验，给出了一些关于如何选择转移任务的有益建议。通过误差分析，还显示了基于 BERT 的模型在故事结尾预测中的优势和劣势。