Large Language Models (LLMs) have exhibited an impressive ability to perform
In-Context Learning (ICL) from only a few examples. Recent works have indicated
that the functions learned by ICL can be represented through compressed vectors
derived from the transformer. However, the working mechanisms and optimization
of these vectors are yet to be thoroughly explored. In this paper, we address
this gap by presenting a comprehensive analysis of these compressed vectors,
drawing parallels to the parameters trained with gradient descent, and
introduce the concept of state vector. Inspired by the works on model soup and
momentum-based gradient descent, we propose inner and momentum optimization
methods that are applied to refine the state vector progressively as test-time
adaptation. Moreover, we simulate state vector aggregation in the multiple
example setting, where demonstrations comprising numerous examples are usually
too lengthy for regular ICL, and further propose a divide-and-conquer
aggregation method to address this challenge. We conduct extensive experiments
using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The
experimental results show that our optimization method effectively enhances the
state vector and achieves the state-of-the-art performance on diverse tasks.
Code is available at this https URL

通过对压缩向量进行细致分析并引入状态向量的概念，我们提出了内部和动量优化方法，并通过模拟状态向量聚合和分割以在多示例情况下进行更好的优化，实验证明这些优化方法有效地提升了状态向量，并在各种任务中取得了最先进的性能。

内外优化的上下文学习状态向量

In-Context Learning State Vector with Inner and Momentum Optimization

High-quality instance segmentation has shown emerging importance in computer
vision. Without any refinement, DCT-Mask directly generates high-resolution
masks by compressed vectors. To further refine masks obtained by compressed
vectors, we propose for the first time a compressed vector based multi-stage
refinement framework. However, the vanilla combination does not bring
significant gains, because changes in some elements of the DCT vector will
affect the prediction of the entire mask. Thus, we propose a simple and novel
method named PatchDCT, which separates the mask decoded from a DCT vector into
several patches and refines each patch by the designed classifier and
regressor. Specifically, the classifier is used to distinguish mixed patches
from all patches, and to correct previously mispredicted foreground and
background patches. In contrast, the regressor is used for DCT vector
prediction of mixed patches, further refining the segmentation quality at
boundary locations. Experiments on COCO show that our method achieves 2.0%,
3.2%, 4.5% AP and 3.4%, 5.3%, 7.0% Boundary AP improvements over Mask-RCNN on
COCO, LVIS, and Cityscapes, respectively. It also surpasses DCT-Mask by 0.7%,
1.1%, 1.3% AP and 0.9%, 1.7%, 4.2% Boundary AP on COCO, LVIS and Cityscapes.
Besides, the performance of PatchDCT is also competitive with other
state-of-the-art methods.

为了进一步完善压缩向量得到的蒙版，我们提出了一种基于压缩向量的多阶段完善框架，并引入了 PatchDCT 方法，通过设计的分类器和回归器对蒙板进行分块、纠正、改善其细节，实验证明该方法在图像识别领域表现良好。