Large Language Models (LLMs) have exhibited an impressive ability to perform
In-Context Learning (ICL) from only a few examples. Recent works have indicated
that the functions learned by ICL can be represented through compressed vectors
derived from the transformer. However, the working mechanisms and optimization
of these vectors are yet to be thoroughly explored. In this paper, we address
this gap by presenting a comprehensive analysis of these compressed vectors,
drawing parallels to the parameters trained with gradient descent, and
introduce the concept of state vector. Inspired by the works on model soup and
momentum-based gradient descent, we propose inner and momentum optimization
methods that are applied to refine the state vector progressively as test-time
adaptation. Moreover, we simulate state vector aggregation in the multiple
example setting, where demonstrations comprising numerous examples are usually
too lengthy for regular ICL, and further propose a divide-and-conquer
aggregation method to address this challenge. We conduct extensive experiments
using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The
experimental results show that our optimization method effectively enhances the
state vector and achieves the state-of-the-art performance on diverse tasks.
Code is available at this https URL

通过对压缩向量进行细致分析并引入状态向量的概念，我们提出了内部和动量优化方法，并通过模拟状态向量聚合和分割以在多示例情况下进行更好的优化，实验证明这些优化方法有效地提升了状态向量，并在各种任务中取得了最先进的性能。