Robotic Process Automation (RPA) systems face challenges in handling complex
processes and diverse screen layouts that require advanced human-like
decision-making capabilities. These systems typically rely on pixel-level
encoding through drag-and-drop or automation frameworks such as Selenium to
create navigation workflows, rather than visual understanding of screen
elements. In this context, we present SmartFlow, an AI-based RPA system that
uses pre-trained large language models (LLMs) coupled with deep-learning based
image understanding. Our system can adapt to new scenarios, including changes
in the user interface and variations in input data, without the need for human
intervention. SmartFlow uses computer vision and natural language processing to
perceive visible elements on the graphical user interface (GUI) and convert
them into a textual representation. This information is then utilized by LLMs
to generate a sequence of actions that are executed by a scripting engine to
complete an assigned task. To assess the effectiveness of SmartFlow, we have
developed a dataset that includes a set of generic enterprise applications with
diverse layouts, which we are releasing for research use. Our evaluations on
this dataset demonstrate that SmartFlow exhibits robustness across different
layouts and applications. SmartFlow can automate a wide range of business
processes such as form filling, customer service, invoice processing, and
back-office operations. SmartFlow can thus assist organizations in enhancing
productivity by automating an even larger fraction of screen-based workflows.
The demo-video and dataset are available at
this https URL

智能流 (SmartFlow) 是一种基于人工智能的机器人流程自动化（RPA）系统，它使用预训练的大型语言模型（LLMs）结合深度学习技术的图像理解，通过计算机视觉和自然语言处理来感知图形用户界面（GUI）上的可见元素并转换为文本表示，然后将其用于生成一系列由脚本引擎执行的操作，以完成所指定的任务。