Large language models have exhibited intriguing in-context learning
capability, achieving promising zero- and few-shot performance without updating
the parameters. However, conventional in-context learning is usually restricted
by length constraints, rendering it ineffective to absorb supervision from a
large number of examples. In order to go beyond few shots, we introduce
structured prompting that breaks the length limit and scales in-context
learning to thousands of examples. Specifically, demonstration examples are
separately encoded with well-designed position embeddings, and then they are
jointly attended by the test example using a rescaled attention mechanism. So
we can scale the number of exemplars with linear complexity instead of
quadratic complexity with respect to length. Experimental results on a diverse
set of tasks show that our approach improves end-task performance and reduces
evaluation variance over conventional in-context learning as the number of
demonstration examples increases. Code has been released at
this https URL.

通过引入结构化的提示，在线学习中处理更多的实例。

结构化提示：将上下文学习扩展到 1,000 个例子

Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Although pretrained language models (PLMs) can be prompted to perform a wide
range of language tasks, it remains an open question how much this ability
comes from generalizable linguistic representations versus more surface-level
lexical patterns. To test this, we present a structured prompting approach that
can be used to prompt for linguistic structure prediction tasks, allowing us to
perform zero- and few-shot sequence tagging with autoregressive PLMs. We
evaluate this approach on part-of-speech tagging, named entity recognition, and
sentence chunking and demonstrate strong few-shot performance in all cases. We
also find that, though the surface forms of the tags provide some signal,
structured prompting can retrieve linguistic structure even with arbitrary
labels, indicating that PLMs contain this knowledge in a general manner robust
to label choice.

使用结构化提示技术，我们证明了预先训练的语言模型（PLMs）可以在零或少量样本情况下执行标记任务，而不是仅靠表面级别的词汇模式，这表明 PLMs 包含了一种通用的语言学知识表示方式。