In-context Learning (ICL) empowers large language models (LLMs) to adapt to
unseen tasks during inference by prefixing a few demonstration examples prior
to test queries. Despite its versatility, ICL incurs substantial computational
and memory overheads compared to zero-shot learning and is susceptible to the
selection and order of demonstration examples. In this work, we introduce
Implicit In-context Learning (I2CL), an innovative paradigm that addresses the
challenges associated with traditional ICL by absorbing demonstration examples
within the activation space. I2CL first generates a condensed vector
representation, namely a context vector, from the demonstration examples. It
then integrates the context vector during inference by injecting a linear
combination of the context vector and query activations into the model's
residual streams. Empirical evaluation on nine real-world tasks across three
model architectures demonstrates that I2CL achieves few-shot performance with
zero-shot cost and exhibits robustness against the variation of demonstration
examples. Furthermore, I2CL facilitates a novel representation of "task-ids",
enhancing task similarity detection and enabling effective transfer learning.
We provide a comprehensive analysis of I2CL, offering deeper insights into its
mechanisms and broader implications for ICL. The source code is available at:
this https URL

在这项研究中，我们介绍了一种名为 Implicit In-context Learning (I2CL) 的创新范式，通过将演示样例吸收到激活空间中，解决了传统 In-context Learning (ICL) 所面临的挑战。经过实证评估，I2CL 在三种模型架构的九个真实世界任务上实现了接近零成本的少样本性能，并且对演示样例的变化表现出鲁棒性。此外，I2CL 促进了一种名为 “task-ids” 的新颖表示，增强了任务相似性检测能力并实现了有效的迁移学习。

隐式上下文学习

Implicit In-context Learning

Large Language Models (LLMs) have gained the ability to assimilate human
knowledge and facilitate natural language interactions with both humans and
other LLMs. However, despite their impressive achievements, LLMs have not made
significant advancements in the realm of graph machine learning. This
limitation arises because graphs encapsulate distinct relational data, making
it challenging to transform them into natural language that LLMs understand. In
this paper, we bridge this gap with a novel framework, GraphText, that
translates graphs into natural language. GraphText derives a graph-syntax tree
for each graph that encapsulates both the node attributes and inter-node
relationships. Traversal of the tree yields a graph text sequence, which is
then processed by an LLM to treat graph tasks as text generation tasks.
Notably, GraphText offers multiple advantages. It introduces training-free
graph reasoning: even without training on graph data, GraphText with ChatGPT
can achieve on par with, or even surpassing, the performance of
supervised-trained graph neural networks through in-context learning (ICL).
Furthermore, GraphText paves the way for interactive graph reasoning, allowing
both humans and LLMs to communicate with the model seamlessly using natural
language. These capabilities underscore the vast, yet-to-be-explored potential
of LLMs in the domain of graph machine learning.

通过一种新的框架 GraphText，将图形转化为自然语言，无需图数据训练，能够实现与监督训练图神经网络相媲美甚至超越其性能的图推理，同时为人类和大型语言模型提供了与模型无缝交流的方式，突显了大型语言模型在图机器学习领域尚未探索的巨大潜力。