As language models continue to scale, Large Language Models (LLMs) have
exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
solve language tasks by prefixing a few in-context demonstrations (ICDs) as
context. Inspired by these advancements, researchers have extended these
techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
However, applying ICL usually faces two major challenges: 1) using more ICDs
will largely increase the inference time and 2) the performance is sensitive to
the selection of ICDs. These challenges are further exacerbated in LMMs due to
the integration of multiple data types and the combinational complexity of
multimodal ICDs. Recently, to address these challenges, some NLP studies
introduce non-learnable In-Context Vectors (ICVs) which extract useful task
information from ICDs into a single vector and then insert it into the LLM to
help solve the corresponding task. However, although useful in simple NLP
tasks, these non-learnable methods fail to handle complex multimodal tasks like
Visual Question Answering (VQA). In this study, we propose \textbf{Learnable
ICV} (L-ICV) to distill essential task information from demonstrations,
improving ICL performance in LMMs. Experiments show that L-ICV can
significantly reduce computational costs while enhancing accuracy in VQA tasks
compared to traditional ICL and other non-learnable ICV methods.

学习型 ICV（L-ICV）从示范中提取必要的任务信息，改善 LMM 中的 ICL 性能，通过与传统 ICL 和其他不可学习的 ICV 方法相比，实验结果显示 L-ICV 能够显著降低计算成本并提高 VQA 任务的准确性。