The pre-training-fine-tuning paradigm based on layout-aware multimodal
pre-trained models has achieved significant progress on document image question
answering. However, domain pre-training and task fine-tuning for additional
visual, layout, and task modules prevent them from directly utilizing
off-the-shelf instruction-tuning language foundation models, which have
recently shown promising potential in zero-shot learning. Contrary to aligning
language models to the domain of document image question answering, we align
document image question answering to off-the-shell instruction-tuning language
foundation models to utilize their zero-shot capability. Specifically, we
propose layout and task aware instruction prompt called LATIN-Prompt, which
consists of layout-aware document content and task-aware descriptions. The
former recovers the layout information among text segments from OCR tools by
appropriate spaces and line breaks. The latter ensures that the model generates
answers that meet the requirements, especially format requirements, through a
detailed description of task. Experimental results on three benchmarks show
that LATIN-Prompt can improve the zero-shot performance of instruction-tuning
language foundation models on document image question answering and help them
achieve comparable levels to SOTAs based on the pre-training-fine-tuning
paradigm. Quantitative analysis and qualitative analysis demonstrate the
effectiveness of LATIN-Prompt. We provide the code in supplementary and will
release the code to facilitate future research.

本研究提出了一种基于 instruction-tuning 语言基础模型的 Zero-shot 能力和称为 LATIN-Prompt 的布局及任务感知的指令提示的方法，通过 OCR 工具中适当的空格和换行符从文本段中恢复布局信息，通过具体的任务描述确保模型生成满足格式要求的答案，提高了指令调整语言基础模型在文档图像问答上的零 - shot 性能，表现与基于预训练 - 微调范例的最新技术水平相当。