Smartphone users often navigate across multiple applications (apps) to
complete tasks such as sharing content between social media platforms.
Autonomous Graphical User Interface (GUI) navigation agents can enhance user
experience in communication, entertainment, and productivity by streamlining
workflows and reducing manual intervention. However, prior GUI agents often
trained with datasets comprising simple tasks that can be completed within a
single app, leading to poor performance in cross-app navigation. To address
this problem, we introduce GUI Odyssey, a comprehensive dataset for training
and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
history resampling module. Extensive experiments demonstrate OdysseyAgent's
superior accuracy compared to existing models. For instance, OdysseyAgent
surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
The dataset and code will be released in
https://github.com/OpenGVLab/GUI-Odyssey.

智能手机用户在不同应用程序之间导航，而自动图形用户界面（GUI）导航代理可以通过简化工作流程和减少手动干预来提高用户在沟通、娱乐和生产力方面的体验。本研究介绍了 GUI Odyssey，这是一个用于训练和评估跨应用程序导航代理的综合数据集。通过利用 GUI Odyssey，研究人员开发了 OdysseyAgent，一个多模式跨应用程序导航代理，其在准确性方面超越了现有模型。