Multi-modal large language models (LLMs) have shown remarkable performance in
various natural language processing tasks, including data extraction from
documents. However, the accuracy of these models can be significantly affected
by document in-plane rotation, also known as skew, a common issue in real-world
scenarios for scanned documents. This study investigates the impact of document
skew on the data extraction accuracy of three state-of-the-art multi-modal
LLMs: Anthropic Claude V3 Sonnet, GPT-4-Turbo, and Llava:v1.6. We focus on
extracting specific entities from synthetically generated sample documents with
varying degrees of skewness. The results demonstrate that document skew
adversely affects the data extraction accuracy of all the tested LLMs, with the
severity of the impact varying across models. We identify the safe in-plane
rotation angles (SIPRA) for each model and investigate the effects of skew on
model hallucinations. Furthermore, we explore existing skew detection and
correction mechanisms and discuss their potential limitations. We propose
alternative approaches, including developing new multi-modal architectures that
are inherently more robust to document skew and incorporating skewing
techniques during the pre-training phase of the models. Additionally, we
highlight the need for more comprehensive testing on a wider range of document
quality and conditions to fully understand the challenges and opportunities
associated with using multi-modal LLMs for information extraction in real-world
scenarios.

探究多模态大型语言模型在文档中存在的旋转问题对数据提取准确性的影响，鉴别出各模型的安全平面旋转角，并研究旋转对模型幻觉的影响，探索现有的旋转检测与修正机制，并提出了包括开发更具鲁棒性的多模态架构和在模型预训练阶段引入旋转技术的替代方法，同时强调有关使用多模态大型语言模型进行信息提取的真实场景中的挑战和机遇需进行更全面的测试。