When pre-trained on large unsupervised textual corpora, language models are
able to store and retrieve factual knowledge to some extent, making it possible
to use them directly for zero-shot cloze-style question answering. However,
storing factual knowledge in a fixed number of weights of a language model
clearly has limitations. Previous approaches have successfully provided access
to information outside the model weights using supervised architectures that
combine an information retrieval system with a machine reading component. In
this paper, we go a step further and integrate information from a retrieval
system with a pre-trained language model in a purely unsupervised way. We
report that augmenting pre-trained language models in this way dramatically
improves performance and that the resulting system, despite being unsupervised,
is competitive with a supervised machine reading baseline. Furthermore,
processing query and context with different segment tokens allows BERT to
utilize its Next Sentence Prediction pre-trained classifier to determine
whether the context is relevant or not, substantially improving BERT's
zero-shot cloze-style question-answering performance and making its predictions
robust to noisy contexts.

本论文研究了在大规模无监督文本语料库上预训练的语言模型，结合信息检索系统以纯无监督方式来增强预训练语言模型，并成功提高了其零样本缺失式问答系统的性能，尤其通过使用不同的分隔符处理查询和上下文，让 BERT 能够更好的判断是否匹配上下文从而增加准确性和鲁棒性。