Exploring the application of powerful large language models (LLMs) on the
fundamental named entity recognition (NER) task has drawn much attention
recently. This work aims to investigate the possibilities of pushing the
boundary of zero-shot NER with LLM via a training-free self-improving strategy.
We propose a self-improving framework, which utilize an unlabeled corpus to
stimulate the self-learning ability of LLMs on NER. First, we use LLM to make
predictions on the unlabeled corpus and obtain the self-annotated data. Second,
we explore various strategies to select reliable samples from the
self-annotated dataset as demonstrations, considering the similarity, diversity
and reliability of demonstrations. Finally, we conduct inference for the test
query via in-context learning with the selected self-annotated demonstrations.
Through comprehensive experimental analysis, our study yielded the following
findings: (1) The self-improving framework further pushes the boundary of
zero-shot NER with LLMs, and achieves an obvious performance improvement; (2)
Iterative self-improving or naively increasing the size of unlabeled corpus
does not guarantee improvements; (3) There might still be space for improvement
via more advanced strategy for reliable entity selection.

通过自我改进策略研究了零样本实体识别任务中大规模语言模型的应用，并提出了利用无标注语料库刺激模型自学能力的自我改进框架，通过全面实验分析发现，该自我改进框架能够进一步推动零样本实体识别任务的发展，并取得了显著的性能提升。

利用大型语言模型进行零样本命名实体识别的自我提升

Self-Improving for Zero-Shot Named Entity Recognition with Large  Language Models

Domain-adaptive pre-training (or DA-training for short), also known as
post-training, aims to train a pre-trained general-purpose language model (LM)
using an unlabeled corpus of a particular domain to adapt the LM so that
end-tasks in the domain can give improved performances. However, existing
DA-training methods are in some sense blind as they do not explicitly identify
what knowledge in the LM should be preserved and what should be changed by the
domain corpus. This paper shows that the existing methods are suboptimal and
proposes a novel method to perform a more informed adaptation of the knowledge
in the LM by (1) soft-masking the attention heads based on their importance to
best preserve the general knowledge in the LM and (2) contrasting the
representations of the general and the full (both general and domain knowledge)
to learn an integrated representation with both general and domain-specific
knowledge. Experimental results will demonstrate the effectiveness of the
proposed approach.

本文提出了一种新的领域适应预训练方法，通过软遮罩注意力头并对一般和完全表示进行对比学习，以更智能的方式对 LM 中的知识进行适应，实验结果表明了该方法的有效性。