Auto-GPT is an autonomous agent that leverages recent advancements in
adapting Large Language Models (LLMs) for decision-making tasks. While there
has been a growing interest in Auto-GPT stypled agents, questions remain
regarding the effectiveness and flexibility of Auto-GPT in solving real-world
decision-making tasks. Its limited capability for real-world engagement and the
absence of benchmarks contribute to these uncertainties. In this paper, we
present a comprehensive benchmark study of Auto-GPT styled agents in
decision-making tasks that simulate real-world scenarios. Our aim is to gain
deeper insights into this problem and understand the adaptability of GPT-based
agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5,
Claude, and Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we
introduce the Additional Opinions algorithm, an easy and effective method that
incorporates supervised/imitation-based learners into the Auto-GPT scheme. This
approach enables lightweight supervised learning without requiring fine-tuning
of the foundational LLMs. We demonstrate through careful baseline comparisons
and ablation studies that the Additional Opinions algorithm significantly
enhances performance in online decision-making benchmarks, including WebShop
and ALFWorld.

本文介绍了一个 Auto-GPT 样式的决策制定任务的综合基准研究，比较了 GPT-4、GPT-3.5、Claude 和 Vicuna 等流行的 LLMs 的性能，并介绍了一种称为 Additional Opinions 算法的易于实现的学习方法，可显著提高在线决策基准测试的性能，包括 WebShop 和 ALFWorld。