Recent advances in Large Vision-Language Models (LVLMs) have significantly
improve performance in image comprehension tasks, such as formatted charts and
rich-content images. Yet, Graphical User Interface (GUI) pose a greater
challenge due to their structured format and detailed textual information.
Existing LVLMs often overly depend on internal knowledge and neglect image
content, resulting in hallucinations and incorrect responses in GUI
comprehension.To address these issues, we introduce VGA, a fine-tuned model
designed for comprehensive GUI understanding. Our model aims to enhance the
interpretation of visual data of GUI and reduce hallucinations. We first
construct a Vision Question Answering (VQA) dataset of 63.8k high-quality
examples with our propose Referent Method, which ensures the model's responses
are highly depend on visual content within the image. We then design a
two-stage fine-tuning method called Foundation and Advanced Comprehension (FAC)
to enhance both the model's ability to extract information from image content
and alignment with human intent. Experiments show that our approach enhances
the model's ability to extract information from images and achieves
state-of-the-art results in GUI understanding tasks. Our dataset and
fine-tuning script will be released soon.

我们提出了一种名为 VGA 的细调模型，旨在提升对图形用户界面（GUI）的理解能力，减少幻觉和错误响应。我们构建了一个 63.8k 高质量示例的 Vision Question Answering（VQA）数据集，并设计了一种名为 Foundation and Advanced Comprehension（FAC）的两阶段细调方法来增强对图像内容的信息提取能力和与人类意图的对齐。实验证明我们的方法提高了从图像中提取信息的能力，并在 GUI 理解任务中取得了最新的成果。我们的数据集和细调脚本将很快发布。