Large Language Models (LLM) have revolutionized Natural Language Processing
(NLP), improving state-of-the-art on many existing tasks and exhibiting
emergent capabilities. However, LLMs have not yet been successfully applied on
semi-structured document information extraction, which is at the core of many
document processing workflows and consists of extracting key entities from a
visually rich document (VRD) given a predefined target schema. The main
obstacles to LLM adoption in that task have been the absence of layout encoding
within LLMs, critical for a high quality extraction, and the lack of a
grounding mechanism ensuring the answer is not hallucinated. In this paper, we
introduce Language Model-based Document Information Extraction and Localization
(LMDX), a methodology to adapt arbitrary LLMs for document information
extraction. LMDX can do extraction of singular, repeated, and hierarchical
entities, both with and without training data, while providing grounding
guarantees and localizing the entities within the document. In particular, we
apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
setting a new state-of-the-art and showing how LMDX enables the creation of
high quality, data-efficient parsers.

基于语言模型的文档信息提取与定位 (LMDX) 方法应用于 PaLM 2-S 模型，通过引入布局编码和保证准确性的基础机制，从视觉丰富的文档中提取出奇异、重复和分层实体，为创建高质量、高效数据分析器创造了新的技术突破。

LMDX：基于语言模型的文件信息提取与定位

LMDX: Language Model-based Document Information Extraction and  Localization

We show that for human-object interaction detection a relatively simple
factorized model with appearance and layout encodings constructed from
pre-trained object detectors outperforms more sophisticated approaches. Our
model includes factors for detection scores, human and object appearance, and
coarse (box-pair configuration) and optionally fine-grained layout (human
pose). We also develop training techniques that improve learning efficiency by:
(1) eliminating a train-inference mismatch; (2) rejecting easy negatives during
mini-batch training; and (3) using a ratio of negatives to positives that is
two orders of magnitude larger than existing approaches. We conduct a thorough
ablation study to understand the importance of different factors and training
techniques using the challenging HICO-Det dataset.

本研究发现，在人物 - 物体交互检测方面，使用预训练的物体检测器构建的外观和布局编码的相对简单的分解模型优于更复杂的方法。同时，作者还开发了一些训练技巧来提高学习效率。研究使用现有的 HICO-Det 数据集展开了深入的剖析，以了解不同因素和培训技术的重要性。