We present Pre-trained Machine Reader (PMR), a novel method to retrofit
Pre-trained Language Models (PLMs) into Machine Reading Comprehension (MRC)
models without acquiring labeled data. PMR is capable of resolving the
discrepancy between model pre-training and downstream fine-tuning of existing
PLMs, and provides a unified solver for tackling various extraction tasks. To
achieve this, we construct a large volume of general-purpose and high-quality
MRC-style training data with the help of Wikipedia hyperlinks and design a Wiki
Anchor Extraction task to guide the MRC-style pre-training process. Although
conceptually simple, PMR is particularly effective in solving extraction tasks
including Extractive Question Answering and Named Entity Recognition, where it
shows tremendous improvements over previous approaches especially under
low-resource settings. Moreover, viewing sequence classification task as a
special case of extraction task in our MRC formulation, PMR is even capable to
extract high-quality rationales to explain the classification process,
providing more explainability of the predictions.

我们提出了 Pre-trained Machine Reader (PMR)，这是一种新方法，旨在将预训练的语言模型（PLMs）转换为机器阅读理解（MRC）模型，而无需获取标记数据， 显示出在包括提取问题回答和命名实体识别在内的提取任务上的显着改进。