Information extraction (IE) is a fundamental area in natural language
processing where prompting large language models (LLMs), even with in-context
examples, cannot defeat small LMs tuned on very small IE datasets. We observe
that IE tasks, such as named entity recognition and relation extraction, all
focus on extracting important information, which can be formalized as a
label-to-span matching. In this paper, we propose a novel framework MetaIE to
build a small LM as meta-model by learning to extract "important information",
i.e., the meta-understanding of IE, so that this meta-model can be adapted to
all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains
the small LM via a symbolic distillation from an LLM following the
label-to-span scheme. We construct the distillation dataset via sampling
sentences from language model pre-training datasets (e.g., OpenWebText in our
implementation) and prompting an LLM to identify the typed spans of "important
information". We evaluate the meta-model under the few-shot adaptation setting.
Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer
a better starting point for few-shot tuning on IE datasets and outperform other
meta-models from (1) vanilla language model pre-training, (2) multi-IE-task
pre-training with human annotations, and (3) single-IE-task symbolic
distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE,
such as the size of the distillation dataset, the meta-model architecture, and
the size of the meta-model.

我们提出了一个称为 MetaIE 的新框架，通过学习提取 “重要信息” 来构建一个小型元模型，使得这个元模型可以有效和高效地适应各种信息提取任务，在少样本适应设置下获得更好的性能。