In this paper, we introduce GUIDE, a novel dataset tailored for the
advancement of Multimodal Large Language Model (MLLM) applications,
particularly focusing on Robotic Process Automation (RPA) use cases. Our
dataset encompasses diverse data from various websites including
Apollo(62.67\%), Gmail(3.43\%), Calendar(10.98\%) and Canva(22.92\%). Each data
entry includes an image, a task description, the last action taken, CoT and the
next action to be performed along with grounding information of where the
action needs to be executed. The data is collected using our in-house advanced
annotation tool NEXTAG (Next Action Grounding and Annotation Tool). The data is
adapted for multiple OS, browsers and display types. It is collected by
multiple annotators to capture the variation of design and the way person uses
a website.
Through this dataset, we aim to facilitate research and development in the
realm of LLMs for graphical user interfaces, particularly in tasks related to
RPA. The dataset's multi-platform nature and coverage of diverse websites
enable the exploration of cross-interface capabilities in automation tasks. We
believe that our dataset will serve as a valuable resource for advancing the
capabilities of multi-platform LLMs in practical applications, fostering
innovation in the field of automation and natural language understanding. Using
GUIDE, we build V-Zen, the first RPA model to automate multiple websites using
our in-House Automation tool AUTONODE

通过介绍 GUIDE 数据集，本文旨在促进多模态大型语言模型 (MLLM) 在图形用户界面领域的研究和开发，特别是与自动化过程中的机器人的使用案例相关的任务。本数据集的多平台特性和多样化的网站覆盖范围使得探索自动化任务中的跨界面能力成为可能。我们相信我们的数据集将成为提升多平台 LLMs 在实际应用中能力、促进自动化和自然语言理解领域创新的宝贵资源。使用 GUIDE 数据集，我们构建了 V-Zen，这是第一个使用我们的内部自动化工具 AUTONODE 在多个网站上进行自动化的 RPA 模型。