Autonomous agents that interact with graphical user interfaces (GUIs) hold
significant potential for enhancing user experiences. To further improve these
experiences, agents need to be personalized and proactive. By effectively
comprehending user intentions through their actions and interactions with GUIs,
agents will be better positioned to achieve these goals. This paper introduces
the task of goal identification from observed UI trajectories, aiming to infer
the user's intended task based on their GUI interactions. We propose a novel
evaluation metric to assess whether two task descriptions are paraphrases
within a specific UI environment. By Leveraging the inverse relation with the
UI automation task, we utilized the Android-In-The-Wild and Mind2Web datasets
for our experiments. Using our metric and these datasets, we conducted several
experiments comparing the performance of humans and state-of-the-art models,
specifically GPT-4 and Gemini-1.5 Pro. Our results show that Gemini performs
better than GPT but still underperforms compared to humans, indicating
significant room for improvement.

介绍了从观察到的用户界面轨迹中抽取目标的任务，并提出了一种新的评估指标来评估特定用户界面环境下两个任务描述是否是释义关系。通过与用户界面自动化任务的逆关系，利用 Android-In-The-Wild 和 Mind2Web 数据集进行了实验，与人类和最先进的模型（GPT-4 和 Gemini-1.5 Pro）进行了比较，结果显示 Gemini 的表现优于 GPT 但仍不及人类，表明有很大的改进空间。