Training models with longer in-context lengths is a significant challenge for
multimodal model due to substantial GPU memory and computational costs. This
exploratory study does not present state-of-the-art models; rather, it
introduces an innovative method designed to increase in-context text length in
multi-modality large language models (MLLMs) efficiently. We present Visualized
In-Context Text Processing (VisInContext), which processes long in-context text
using visual tokens. This technique significantly reduces GPU memory usage and
floating point operations (FLOPs) for both training and inferenceing stage. For
instance, our method expands the pre-training in-context text length from 256
to 2048 tokens with nearly same FLOPs for a 56 billion parameter MOE model.
Experimental results demonstrate that model trained with VisInContext delivers
superior performance on common downstream benchmarks for in-context few-shot
evaluation. Additionally, VisInContext is complementary to existing methods for
increasing in-context text length and enhances document understanding
capabilities, showing great potential in document QA tasks and sequential
document retrieval.

使用 Visualized In-Context Text Processing (VisInContext) 技术能够有效地增加多模态大型语言模型中的上下文文本长度，减少 GPU 内存使用和浮点运算，同时在文档理解任务和顺序文档检索中展示了巨大的潜力。