Intrinsically motivated exploration has proven useful for reinforcement
learning, even without additional extrinsic rewards. When the environment is
naturally represented as a graph, how to guide exploration best remains an open
question. In this work, we propose a novel approach for exploring
graph-structured data motivated by two theories of human curiosity: the
information gap theory and the compression progress theory. The theories view
curiosity as an intrinsic motivation to optimize for topological features of
subgraphs induced by the visited nodes in the environment. We use these
proposed features as rewards for graph neural-network-based reinforcement
learning. On multiple classes of synthetically generated graphs, we find that
trained agents generalize to larger environments and to longer exploratory
walks than are seen during training. Our method computes more efficiently than
the greedy evaluation of the relevant topological properties. The proposed
intrinsic motivations bear particular relevance for recommender systems. We
demonstrate that curiosity-based recommendations are more predictive of human
behavior than PageRank centrality for several real-world graph datasets,
including MovieLens, Amazon Books, and Wikispeedia.

通过结合人类好奇心理论，使用基于图神经网络的强化学习来探索图形化数据，这种内部动机探索方法尤其适用于推荐系统

利用人类好奇心的网络理论进行本质动机的图形探索

Intrinsically motivated graph exploration using network theories of  human curiosity

Reinforcement learning algorithms typically struggle in the absence of a
dense, well-shaped reward function. Intrinsically motivated exploration methods
address this limitation by rewarding agents for visiting novel states or
transitions, but these methods offer limited benefits in large environments
where most discovered novelty is irrelevant for downstream tasks. We describe a
method that uses background knowledge from text corpora to shape exploration.
This method, called ELLM (Exploring with LLMs) rewards an agent for achieving
goals suggested by a language model prompted with a description of the agent's
current state. By leveraging large-scale language model pretraining, ELLM
guides agents toward human-meaningful and plausibly useful behaviors without
requiring a human in the loop. We evaluate ELLM in the Crafter game environment
and the Housekeep robotic simulator, showing that ELLM-trained agents have
better coverage of common-sense behaviors during pretraining and usually match
or improve performance on a range of downstream tasks.

提出了一种名为 ELLM（LLM 探索）的方法，它利用来自文本语料库的背景知识来塑造探索，通过利用大规模语言模型预训练，无需人介入便能引导智能体朝向有人类意义的和可能有用的行为方向，通过在 Crafter 游戏环境和 Housekeep 机器人模拟器中的实验，证明了 ELLM 训练的代理在预训练期间具有更好的常识行为覆盖，并且在一系列下游任务中通常与或优于性能。

利用大型语言模型指导强化学习的预训练

Guiding Pretraining in Reinforcement Learning with Large Language Models

In open-ended environments, autonomous learning agents must set their own
goals and build their own curriculum through an intrinsically motivated
exploration. They may consider a large diversity of goals, aiming to discover
what is controllable in their environments, and what is not. Because some goals
might prove easy and some impossible, agents must actively select which goal to
practice at any moment, to maximize their overall mastery on the set of
learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a
modular Universal Value Function Approximator with hindsight learning to
achieve a diversity of goals of different kinds within a unique policy and 2)
an automated curriculum learning mechanism that biases the attention of the
agent towards goals maximizing the absolute learning progress. Agents focus
sequentially on goals of increasing complexity, and focus back on goals that
are being forgotten. Experiments conducted in a new modular-goal robotic
environment show the resulting developmental self-organization of a learning
curriculum, and demonstrate properties of robustness to distracting goals,
forgetting and changes in body properties.

该论文提出了一种名为 CURIOUS 的算法，它利用 MODULAR Universal Value Function Approximator 和自动化课程学习机制来实现学习代理的自主目标设定和自我组织学习课程，实现学习目标的快速最优化。

CURIOUS: 内在动机模块化多目标强化学习

CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement  Learning

Intrinsically motivated spontaneous exploration is a key enabler of
autonomous developmental learning in human children. It enables the discovery
of skill repertoires through autotelic learning, i.e. the self-generation,
self-selection, self-ordering and self-experimentation of learning goals. We
present an algorithmic approach called Intrinsically Motivated Goal Exploration
Processes (IMGEP) to enable similar properties of autonomous learning in
machines. The IMGEP architecture relies on several principles: 1)
self-generation of goals, generalized as parameterized fitness functions; 2)
selection of goals based on intrinsic rewards; 3) exploration with incremental
goal-parameterized policy search and exploitation with a batch learning
algorithm; 4) systematic reuse of information acquired when targeting a goal
for improving towards other goals. We present a particularly efficient form of
IMGEP, called AMB, that uses a population-based policy and an object-centered
spatio-temporal modularity. We provide several implementations of this
architecture and demonstrate their ability to automatically generate a learning
curriculum within several experimental setups. One of these experiments
includes a real humanoid robot exploring multiple spaces of goals with several
hundred continuous dimensions and with distractors. While no particular target
goal is provided to these autotelic agents, this curriculum allows the
discovery of diverse skills that act as stepping stones for learning more
complex skills, e.g. nested tool use.

本论文提出了一种基于内在动机的目标探索算法，结合自主学习过程和增量式目标策略搜寻探索，可以自动生成任务序列并发掘多种技能，无需特定的目标指令，适用于机器人等人工智能设备的自主发展学习。