In-context Learning (ICL) empowers large language models (LLMs) to adapt to
unseen tasks during inference by prefixing a few demonstration examples prior
to test queries. Despite its versatility, ICL incurs substantial computational
and memory overheads compared to zero-shot learning and is susceptible to the
selection and order of demonstration examples. In this work, we introduce
Implicit In-context Learning (I2CL), an innovative paradigm that addresses the
challenges associated with traditional ICL by absorbing demonstration examples
within the activation space. I2CL first generates a condensed vector
representation, namely a context vector, from the demonstration examples. It
then integrates the context vector during inference by injecting a linear
combination of the context vector and query activations into the model's
residual streams. Empirical evaluation on nine real-world tasks across three
model architectures demonstrates that I2CL achieves few-shot performance with
zero-shot cost and exhibits robustness against the variation of demonstration
examples. Furthermore, I2CL facilitates a novel representation of "task-ids",
enhancing task similarity detection and enabling effective transfer learning.
We provide a comprehensive analysis of I2CL, offering deeper insights into its
mechanisms and broader implications for ICL. The source code is available at:
this https URL

在这项研究中，我们介绍了一种名为 Implicit In-context Learning (I2CL) 的创新范式，通过将演示样例吸收到激活空间中，解决了传统 In-context Learning (ICL) 所面临的挑战。经过实证评估，I2CL 在三种模型架构的九个真实世界任务上实现了接近零成本的少样本性能，并且对演示样例的变化表现出鲁棒性。此外，I2CL 促进了一种名为 “task-ids” 的新颖表示，增强了任务相似性检测能力并实现了有效的迁移学习。