This paper explores capabilities of Vision Language Models on spreadsheet
comprehension. We propose three self-supervised challenges with corresponding
evaluation metrics to comprehensively evaluate VLMs on Optical Character
Recognition (OCR), spatial perception, and visual format recognition.
Additionally, we utilize the spreadsheet table detection task to assess the
overall performance of VLMs by integrating these challenges. To probe VLMs more
finely, we propose three spreadsheet-to-image settings: column width
adjustment, style change, and address augmentation. We propose variants of
prompts to address the above tasks in different settings. Notably, to leverage
the strengths of VLMs in understanding text rather than two-dimensional
positioning, we propose to decode cell values on the four boundaries of the
table in spreadsheet boundary detection. Our findings reveal that VLMs
demonstrate promising OCR capabilities but produce unsatisfactory results due
to cell omission and misalignment, and they notably exhibit insufficient
spatial and format recognition skills, motivating future work to enhance VLMs'
spreadsheet data comprehension capabilities using our methods to generate
extensive spreadsheet-image pairs in various settings.

本文通过在电子表格理解方面探索视觉语言模型的能力，提出了三个自我学习挑战并伴随相应的评估方法，全面评估了 VLMs 在光学字符识别、空间感知和视觉格式识别等方面的能力。同时，还提出三种电子表格到图像的设定，以更细致地探索 VLMs，其中包括列宽调整、样式变化和地址扩充。研究结果表明，VLMs 在 OCR 方面表现出有前途的能力，但由于单元格遗漏和错位，结果不尽人意；同时，它们在空间感知和格式识别技巧方面明显不足，为了增强 VLMs 对电子表格数据的理解能力，我们提出的方法可以在各种设置下生成大量的电子表格 - 图像对，值得进一步探索。