Modern LVLMs still struggle to achieve fine-grained document understanding,
such as OCR/translation/caption for regions of interest to the user, tasks that
require the context of the entire page, or even multiple pages. Accordingly,
this paper proposes Fox, an effective pipeline, hybrid data, and tuning
strategy, that catalyzes LVLMs to focus anywhere on single/multi-page
documents. We introduce a novel task to boost the document understanding by
making LVLMs focus attention on the document-level region, such as redefining
full-page OCR as foreground focus. We employ multiple vision vocabularies to
extract visual hybrid knowledge for interleaved document pages (e.g., a page
containing a photo). Meanwhile, we render cross-vocabulary vision data as the
catalyzer to achieve a full reaction of multiple visual vocabularies and
in-document figure understanding. Further, without modifying the weights of
multiple vision vocabularies, the above catalyzed fine-grained understanding
capabilities can be efficiently tuned to multi-page documents, enabling the
model to focus anywhere in both format-free and page-free manners. Besides, we
build a benchmark including 9 fine-grained sub-tasks (e.g., region-level
OCR/summary, color-guided OCR) to promote document analysis in the community.
The experimental results verify the superiority of our model.

本文提出了一种名为 Fox 的高效流水线、混合数据和调优策略，以推动 LVLM 在单页 / 多页文档上的细粒度文档理解，引入新颖任务以提升文档理解，并通过多模式视觉词汇和跨视觉数据的应用来实现多种视觉词汇和文档内图像理解的全面反应。