Autonomous agents that interact with graphical user interfaces (GUIs) hold
significant potential for enhancing user experiences. To further improve these
experiences, agents need to be personalized and proactive. By effectively
comprehending user intentions through their actions and interactions with GUIs,
agents will be better positioned to achieve these goals. This paper introduces
the task of goal identification from observed UI trajectories, aiming to infer
the user's intended task based on their GUI interactions. We propose a novel
evaluation metric to assess whether two task descriptions are paraphrases
within a specific UI environment. By Leveraging the inverse relation with the
UI automation task, we utilized the Android-In-The-Wild and Mind2Web datasets
for our experiments. Using our metric and these datasets, we conducted several
experiments comparing the performance of humans and state-of-the-art models,
specifically GPT-4 and Gemini-1.5 Pro. Our results show that Gemini performs
better than GPT but still underperforms compared to humans, indicating
significant room for improvement.

介绍了从观察到的用户界面轨迹中抽取目标的任务，并提出了一种新的评估指标来评估特定用户界面环境下两个任务描述是否是释义关系。通过与用户界面自动化任务的逆关系，利用 Android-In-The-Wild 和 Mind2Web 数据集进行了实验，与人类和最先进的模型（GPT-4 和 Gemini-1.5 Pro）进行了比较，结果显示 Gemini 的表现优于 GPT 但仍不及人类，表明有很大的改进空间。

从用户界面轨迹识别用户目标

Identifying User Goals from UI Trajectories

Designing agents, capable of learning autonomously a wide range of skills is
critical in order to increase the scope of reinforcement learning. It will both
increase the diversity of learned skills and reduce the burden of manually
designing reward functions for each skill. Self-supervised agents, setting
their own goals, and trying to maximize the diversity of those goals have shown
great promise towards this end. However, a currently known limitation of agents
trying to maximize the diversity of sampled goals is that they tend to get
attracted to noise or more generally to parts of the environments that cannot
be controlled (distractors). When agents have access to predefined goal
features or expert knowledge, absolute Learning Progress (ALP) provides a way
to distinguish between regions that can be controlled and those that cannot.
However, those methods often fall short when the agents are only provided with
raw sensory inputs such as images. In this work we extend those concepts to
unsupervised image-based goal exploration. We propose a framework that allows
agents to autonomously identify and ignore noisy distracting regions while
searching for novelty in the learnable regions to both improve overall
performance and avoid catastrophic forgetting. Our framework can be combined
with any state-of-the-art novelty seeking goal exploration approaches. We
construct a rich 3D image based environment with distractors. Experiments on
this environment show that agents using our framework successfully identify
interesting regions of the environment, resulting in drastically improved
performances. The source code is available at
this https URL

提出了一种框架，可以使代理能够自主识别和忽略干扰区域，从而在可学习区域中寻找新颖性，改善整体表现并避免灾难性遗忘。实验表明，代理利用该框架成功地识别了环境的有趣区域，并在性能上取得了极大的改善。