Building generalist agents that can handle diverse tasks and evolve
themselves across different environments is a long-term goal in the AI
community. Large language models (LLMs) are considered a promising foundation
to build such agents due to their generalized capabilities. Current approaches
either have LLM-based agents imitate expert-provided trajectories step-by-step,
requiring human supervision, which is hard to scale and limits environmental
exploration; or they let agents explore and learn in isolated environments,
resulting in specialist agents with limited generalization. In this paper, we
take the first step towards building generally-capable LLM-based agents with
self-evolution ability. We identify a trinity of ingredients: 1) diverse
environments for agent exploration and learning, 2) a trajectory set to equip
agents with basic capabilities and prior knowledge, and 3) an effective and
scalable evolution method. We propose AgentGym, a new framework featuring a
variety of environments and tasks for broad, real-time, uni-format, and
concurrent agent exploration. AgentGym also includes a database with expanded
instructions, a benchmark suite, and high-quality trajectories across
environments. Next, we propose a novel method, AgentEvol, to investigate the
potential of agent self-evolution beyond previously seen data across tasks and
environments. Experimental results show that the evolved agents can achieve
results comparable to SOTA models. We release the AgentGym suite, including the
platform, dataset, benchmark, checkpoints, and algorithm implementations. The
AgentGym suite is available on this https URL

构建能够处理多样化任务并在不同环境中自我演进的通用智能体是人工智能领域的长期目标。本文提出了 AgentGym，一个新的框架，具备多样的环境和任务，用于广泛、实时、统一格式和并行的智能体探索。并且，我们还提出了一种名为 AgentEvol 的新方法，研究智能体在任务和环境中自我演进的潜力。实验结果表明，演进的智能体能够达到与最先进模型相当的结果。

AgentGym：基于大型语言模型的智能体在多样环境中的进化

AgentGym: Evolving Large Language Model-based Agents across Diverse  Environments

Large Language Models (LLMs) have showcased remarkable versatility across
diverse domains. However, the pathway toward autonomous model development, a
cornerstone for achieving human-level learning and advancing autonomous AI,
remains largely uncharted. We introduce an innovative approach, termed "SELF"
(Self-Evolution with Language Feedback). This methodology empowers LLMs to
undergo continual self-evolution. Furthermore, SELF employs language-based
feedback as a versatile and comprehensive evaluative tool, pinpointing areas
for response refinement and bolstering the stability of self-evolutionary
training. Initiating with meta-skill learning, SELF acquires foundational
meta-skills with a focus on self-feedback and self-refinement. These
meta-skills are critical, guiding the model's subsequent self-evolution through
a cycle of perpetual training with self-curated data, thereby enhancing its
intrinsic abilities. Given unlabeled instructions, SELF equips the model with
the capability to autonomously generate and interactively refine responses.
This synthesized training data is subsequently filtered and utilized for
iterative fine-tuning, enhancing the model's capabilities. Experimental results
on representative benchmarks substantiate that SELF can progressively advance
its inherent abilities without the requirement of human intervention, thereby
indicating a viable pathway for autonomous model evolution. Additionally, SELF
can employ online self-refinement strategy to produce responses of superior
quality. In essence, the SELF framework signifies a progressive step towards
autonomous LLM development, transforming the LLM from a mere passive recipient
of information into an active participant in its own evolution.

使用自我进化与语言反馈的创新方法，自主学习和发展自主人工智能的关键，使大型语言模型在不同领域展现出了非凡的适应能力。

自我驱动的语言模型大规模自我演进

SELF: Language-Driven Self-Evolution for Large Language Model

Self-evolution is indispensable to realize full autonomous driving. This
paper presents a self-evolving decision-making system based on the Integrated
Decision and Control (IDC), an advanced framework built on reinforcement
learning (RL). First, an RL algorithm called constrained mixed policy gradient
(CMPG) is proposed to consistently upgrade the driving policy of the IDC. It
adapts the MPG under the penalty method so that it can solve constrained
optimization problems using both the data and model. Second, an attention-based
encoding (ABE) method is designed to tackle the state representation issue. It
introduces an embedding network for feature extraction and a weighting network
for feature fusion, fulfilling order-insensitive encoding and importance
distinguishing of road users. Finally, by fusing CMPG and ABE, we develop the
first data-driven decision and control system under the IDC architecture, and
deploy the system on a fully-functional self-driving vehicle running in daily
operation. Experiment results show that boosting by data, the system can
achieve better driving ability over model-based methods. It also demonstrates
safe, efficient and smart driving behavior in various complex scenes at a
signalized intersection with real mixed traffic flow.

本文提出了一种基于 RL 算法的自我进化决策系统，包含 CMPG 和 ABE 技术，该系统被部署在一辆自动驾驶汽车上，实现了智能、高效、安全的驾驶行为。实验表明，该系统可显著提升驾驶能力和模型方法相比具有更好的驾驶环境适应性。