The remarkable success of pretrained language models has motivated the study
of what kinds of knowledge these models learn during pretraining. Reformulating
tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach
for gauging such knowledge, however, its usage is limited by the manual effort
and guesswork required to write suitable prompts. To address this, we develop
AutoPrompt, an automated method to create prompts for a diverse set of tasks,
based on a gradient-guided search. Using AutoPrompt, we show that masked
language models (MLMs) have an inherent capability to perform sentiment
analysis and natural language inference without additional parameters or
finetuning, sometimes achieving performance on par with recent state-of-the-art
supervised models. We also show that our prompts elicit more accurate factual
knowledge from MLMs than the manually created prompts on the LAMA benchmark,
and that MLMs can be used as relation extractors more effectively than
supervised relation extraction models. These results demonstrate that
automatically generated prompts are a viable parameter-free alternative to
existing probing methods, and as pretrained LMs become more sophisticated and
capable, potentially a replacement for finetuning.

利用自动化方法生成的 AutoPrompt，我们展示了预训练语言模型在自然语言推理、情感分析和关系提取方面的潜在能力，以及自动生成的提示方法是现有探究方法的一个可行的无参数替代方法。

AutoPrompt: 使用自动生成的提示从语言模型中引导知识

AutoPrompt: Eliciting Knowledge from Language Models with Automatically  Generated Prompts

General purpose relation extractors, which can model arbitrary relations, are
a core aspiration in information extraction. Efforts have been made to build
general purpose extractors that represent relations with their surface forms,
or which jointly embed surface forms with relations from an existing knowledge
graph. However, both of these approaches are limited in their ability to
generalize. In this paper, we build on extensions of Harris' distributional
hypothesis to relations, as well as recent advances in learning text
representations (specifically, BERT), to build task agnostic relation
representations solely from entity-linked text. We show that these
representations significantly outperform previous work on exemplar based
relation extraction (FewRel) even without using any of that task's training
data. We also show that models initialized with our task agnostic
representations, and then tuned on supervised relation extraction datasets,
significantly outperform the previous methods on SemEval 2010 Task 8, KBP37,
and TACRED.

通过建立基于 Harris 的分布式假设的关系模型，以及最近的文本表示学习（特别是 BERT），仅从实体相关的文本中构建任务无关的关系表示，我们排除了 FewRel 的任何训练数据，证明了这些表示在关系提取方面的显著优势。我们还表明，初始化为我们的任务无关表示，然后在监督关系提取数据集上进行调整的模型，在 SemEval 2010 任务 8、KBP37 和 TACRED 上显著优于以前的方法。