Designing effective reward functions is crucial to training reinforcement
learning (RL) algorithms. However, this design is non-trivial, even for domain
experts, due to the subjective nature of certain tasks that are hard to
quantify explicitly. In recent works, large language models (LLMs) have been
used for reward generation from natural language task descriptions, leveraging
their extensive instruction tuning and commonsense understanding of human
behavior. In this work, we hypothesize that LLMs, guided by human feedback, can
be used to formulate human-aligned reward functions. Specifically, we study
this in the challenging setting of autonomous driving (AD), wherein notions of
"good" driving are tacit and hard to quantify. To this end, we introduce
REvolve, an evolutionary framework that uses LLMs for reward design in AD.
REvolve creates and refines reward functions by utilizing human feedback to
guide the evolution process, effectively translating implicit human knowledge
into explicit reward functions for training (deep) RL agents. We demonstrate
that agents trained on REvolve-designed rewards align closely with human
driving standards, thereby outperforming other state-of-the-art baselines.

利用大型语言模型来设计人类对鼓励功能的反馈机制，从而更好地训练自主驾驶系统。

REvolve: 大型语言模型在自动驾驶中的奖励进化

REvolve: Reward Evolution with Large Language Models for Autonomous  Driving

A major challenge in decision making domains with large state spaces is to
effectively select actions which maximize utility. In recent years, approaches
such as reinforcement learning (RL) and search algorithms have been successful
to tackle this issue, despite their differences. RL defines a learning
framework that an agent explores and interacts with. Search algorithms provide
a formalism to search for a solution. However, it is often difficult to
evaluate the performances of such approaches in a practical way. Motivated by
this problem, we focus on one game domain, i.e., Connect-4, and develop a novel
evolutionary framework to evaluate three classes of algorithms: RL, Minimax and
Monte Carlo tree search (MCTS). The contribution of this paper is threefold: i)
we implement advanced versions of these algorithms and provide a systematic
comparison with their standard counterpart, ii) we develop a novel evaluation
framework, which we call the Evolutionary Tournament, and iii) we conduct an
extensive evaluation of the relative performance of each algorithm to compare
our findings. We evaluate different metrics and show that MCTS achieves the
best results in terms of win percentage, whereas Minimax and Q-Learning are
ranked in second and third place, respectively, although the latter is shown to
be the fastest to make a decision.

在大状态空间的决策领域中，有效选择最大效用的行动是一个主要挑战。本文针对一个游戏领域 ——Connect-4，开发了一种新颖的进化框架来评估三类算法：强化学习（RL）、极大极小算法（Minimax）和蒙特卡洛树搜索（MCTS）。研究发现，MCTS 在胜率方面取得了最好的结果，而 Minimax 和 Q-Learning 分别排名第二和第三，尽管后者在决策速度方面表现最快。

演化算法评估 Connect-4 中高级 Minimax、Q-Learning 和 MCTS 的比较框架

An Evolutionary Framework for Connect-4 as Test-Bed for Comparison of  Advanced Minimax, Q-Learning and MCTS

Optimization can be found in many real-life applications. Designing an
effective algorithm for a specific optimization problem typically requires a
tedious amount of effort from human experts with domain knowledge and algorithm
design skills. In this paper, we propose a novel approach called Algorithm
Evolution using Large Language Model (AEL). It utilizes a large language model
(LLM) to automatically generate optimization algorithms via an evolutionary
framework. AEL does algorithm-level evolution without model training. Human
effort and requirements for domain knowledge can be significantly reduced. We
take constructive methods for the salesman traveling problem as a test example,
we show that the constructive algorithm obtained by AEL outperforms simple
hand-crafted and LLM-generated heuristics. Compared with other domain deep
learning model-based algorithms, these methods exhibit excellent scalability
across different problem sizes. AEL is also very different from previous
attempts that utilize LLMs as search operators in algorithms.

利用大型语言模型进行算法进化，能够自动生成优化算法，减少人工专家和领域知识的需求，在解决推销员旅行问题方面表现优异，具有出色的可伸缩性，与先前的利用语言模型作为搜索操作符的尝试大不相同。

使用大型语言模型的算法演进

Algorithm Evolution Using Large Language Model

Influence maximization is a crucial issue for mining the deep information of
social networks, which aims to select a seed set from the network to maximize
the number of influenced nodes. To evaluate the influence spread of a seed set
efficiently, existing studies have proposed transformations with lower
computational costs to replace the expensive Monte Carlo simulation process.
These alternate transformations, based on network prior knowledge, induce
different search behaviors with similar characteristics to various
perspectives. Specifically, it is difficult for users to determine a suitable
transformation a priori. This article proposes a multi-transformation
evolutionary framework for influence maximization (MTEFIM) with convergence
guarantees to exploit the potential similarities and unique advantages of
alternate transformations and to avoid users manually determining the most
suitable one. In MTEFIM, multiple transformations are optimized simultaneously
as multiple tasks. Each transformation is assigned an evolutionary solver.
Three major components of MTEFIM are conducted via: 1) estimating the potential
relationship across transformations based on the degree of overlap across
individuals of different populations, 2) transferring individuals across
populations adaptively according to the inter-transformation relationship, and
3) selecting the final output seed set containing all the transformation's
knowledge. The effectiveness of MTEFIM is validated on both benchmarks and
real-world social networks. The experimental results show that MTEFIM can
efficiently utilize the potentially transferable knowledge across multiple
transformations to achieve highly competitive performance compared to several
popular IM-specific methods. The implementation of MTEFIM can be accessed at
this https URL.

本文提出了一种多变换进化框架来最大化影响，在众多转换方法中利用最适合的策略来选择种子集合，实验结果表明，该方法在多个基准测试和实际社交网络上均表现出高效的性能。