In the rapidly evolving landscape of AI research and application, Multimodal
Large Language Models (MLLMs) have emerged as a transformative force, adept at
interpreting and integrating information from diverse modalities such as text,
images, and Graphical User Interfaces (GUIs). Despite these advancements, the
nuanced interaction and understanding of GUIs pose a significant challenge,
limiting the potential of existing models to enhance automation levels. To
bridge this gap, this paper presents V-Zen, an innovative Multimodal Large
Language Model (MLLM) meticulously crafted to revolutionise the domain of GUI
understanding and grounding. Equipped with dual-resolution image encoders,
V-Zen establishes new benchmarks in efficient grounding and next-action
prediction, thereby laying the groundwork for self-operating computer systems.
Complementing V-Zen is the GUIDE dataset, an extensive collection of real-world
GUI elements and task-based sequences, serving as a catalyst for specialised
fine-tuning. The successful integration of V-Zen and GUIDE marks the dawn of a
new era in multimodal AI research, opening the door to intelligent, autonomous
computing experiences. This paper extends an invitation to the research
community to join this exciting journey, shaping the future of GUI automation.
In the spirit of open science, our code, data, and model will be made publicly
available, paving the way for multimodal dialogue scenarios with intricate and
precise interactions.

V-Zen 是一款创新的多模态大语言模型，通过双分辨率图像编码器在 GUI 理解和下一步行动预测方面取得了突破性的成果，并与 GUIDE 数据集成功集成，标志着多模态人工智能研究的新时代的来临。

V-Zen：高效 GUI 理解与准确引用的新型多模态 LLM

V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel  Multimodal LLM

People are spending an enormous amount of time on digital devices through
graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
language models (LLMs) such as ChatGPT can assist people in tasks like writing
emails, but struggle to understand and interact with GUIs, thus limiting their
potential to increase automation levels. In this paper, we introduce CogAgent,
an 18-billion-parameter visual language model (VLM) specializing in GUI
understanding and navigation. By utilizing both low-resolution and
high-resolution image encoders, CogAgent supports input at a resolution of
1120*1120, enabling it to recognize tiny page elements and text. As a
generalist visual language model, CogAgent achieves the state of the art on
five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
only screenshots as input, outperforms LLM-based methods that consume extracted
HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
advancing the state of the art. The model and codes are available at
https://github.com/THUDM/CogVLM.

本研究介绍了 CogAgent，这是一个在 GUI 理解和导航方面专门使用的 180 亿参数的视觉语言模型，通过低分辨率和高分辨率图像编码器，支持 1120*1120 分辨率的输入，能够识别微小的页面元素和文本。CogAgent 在五个文本丰富和四个通用的 VQA 基准测试中达到了最新的技术水平，在 PC 和 Android 的 GUI 导航任务中超过了 LLM 方法，从而推动了技术的发展。