Large language model (LLM) leads to a surge of autonomous GUI agents for
smartphone, which completes a task triggered by natural language through
predicting a sequence of actions of API. Even though the task highly relies on
past actions and visual observations, existing studies typical consider little
semantic information carried out by intermediate screenshots and screen
operations. To address this, this work presents Chain-of-Action-Thought (dubbed
CoAT), which takes the description of the previous actions, the current screen,
and more importantly the action thinking of what actions should be performed
and the outcomes led by the chosen action. We demonstrate that, in a zero-shot
setting upon an off-the-shell LLM, CoAT significantly improves the goal
progress compared to standard context modeling. To further facilitate the
research in this line, we construct a benchmark Android-In-The-Zoo (AitZ),
which contains 18,643 screen-action pairs together with chain-of-action-thought
annotations. Experiments show that fine-tuning a 200M model on our AitZ dataset
achieves on par performance with CogAgent-Chat-18B.

通过描述以前的操作、当前屏幕和选择操作所导致的结果的操作思考，Chain-of-Action-Thought 架构与大型语言模型相结合，在智能手机上实现了通过自然语言触发的任务完成，显著提高了目标进展。