Understanding visually situated language requires recognizing text and visual
elements, and interpreting complex layouts. State-of-the-art methods commonly
use specialized pre-processing tools, such as optical character recognition
(OCR) systems, that map document image inputs to extracted information in the
space of textual tokens, and sometimes also employ large language models (LLMs)
to reason in text token space. However, the gains from external tools and LLMs
come at the cost of increased computational and engineering complexity. In this
paper, we ask whether small pretrained image-to-text models can learn selective
text or layout recognition and reasoning as an intermediate inference step in
an end-to-end model for pixel-level visual language understanding. We
incorporate the outputs of such OCR tools, LLMs, and larger multimodal models
as intermediate ``rationales'' on training data, and train a small student
model to predict both rationales and answers for input questions based on those
training examples. A student model based on Pix2Struct (282M parameters)
achieves consistent improvements on three visual document understanding
benchmarks representing infographics, scanned documents, and figures, with
improvements of more than 4\% absolute over a comparable Pix2Struct model that
predicts answers directly.

通过小型的预训练图像到文本模型，我们在像素级视觉语言理解的端到端模型中学习选择性文本或布局识别和推理，取得了在信息图表、扫描文档和图像三个视觉文档理解基准中的一致性改进，超过直接预测答案的相似 Pix2Struct 模型 4% 以上。