An automation system that can execute natural language instructions by
driving the user interface (UI) of an application can benefit users, especially
when situationally or permanently impaired. Traditional automation systems
(manual scripting, programming by demonstration tools, etc.) do not produce
generalizable models that can tolerate changes in the UI or task workflow.
Machine-learned automation agents generalize better, but either work only in
simple, hand-crafted applications or rely on large pre-trained models, which
may be too computationally expensive to run on mobile devices. In this paper,
we propose \emph{UINav}, a demonstration-based agent maker system. UINav agents
are lightweight enough to run on mobile devices, yet they achieve high success
rates with a modest number of task demonstrations. To minimize the number of
task demonstrations, UINav includes a referee model that allows users to
receive immediate feedback on tasks where the agent is failing to best guide
efforts to collect additional demonstrations. Further, UINav adopts macro
actions to reduce an agent's state space, and augments human demonstrations to
increase the diversity of training data. Our evaluation demonstrates that with
an average of 10 demonstrations per task UINav can achieve an accuracy of 70\%
or higher, and that with enough demonstrations it can achieve near-perfect
success rates on 40+ different tasks.

提出了一种名为 UINav 的演示式代理系统，能够以简单手势为主的任务演示实现高成功率，并在保持较轻量级的同时可在移动设备上运行，通过采用裁判模型、宏操作和增加训练数据的方式，最小化任务演示次数，对于每个任务平均 10 次演示，UINav 能够达到 70% 以上的准确率，并在足够多演示的情况下实现 40 个以上不同任务的几乎完美成功率。