Effective exploration is crucial to discovering optimal strategies for
multi-agent reinforcement learning (MARL) in complex coordination tasks.
Existing methods mainly utilize intrinsic rewards to enable committed
exploration or use role-based learning for decomposing joint action spaces
instead of directly conducting a collective search in the entire
action-observation space. However, they often face challenges obtaining
specific joint action sequences to reach successful states in long-horizon
tasks. To address this limitation, we propose Imagine, Initialize, and Explore
(IIE), a novel method that offers a promising solution for efficient
multi-agent exploration in complex scenarios. IIE employs a transformer model
to imagine how the agents reach a critical state that can influence each
other's transition functions. Then, we initialize the environment at this state
using a simulator before the exploration phase. We formulate the imagination as
a sequence modeling problem, where the states, observations, prompts, actions,
and rewards are predicted autoregressively. The prompt consists of
timestep-to-go, return-to-go, influence value, and one-shot demonstration,
specifying the desired state and trajectory as well as guiding the action
generation. By initializing agents at the critical states, IIE significantly
increases the likelihood of discovering potentially important under-explored
regions. Despite its simplicity, empirical results demonstrate that our method
outperforms multi-agent exploration baselines on the StarCraft Multi-Agent
Challenge (SMAC) and SMACv2 environments. Particularly, IIE shows improved
performance in the sparse-reward SMAC tasks and produces more effective
curricula over the initialized states than other generative methods, such as
CVAE-GAN and diffusion models.

通过使用一种新的方法，称为 Imagine, Initialize, and Explore (IIE)，该方法使用转换模型使多个智能体达到关键状态，并以此为起点进行探索，实现高效多智能体探索。在实际测试中，IIE 在复杂环境下表现出优异的性能，超过了其他多智能体探索基线，特别在稀疏奖励任务中有更好的表现，并产生比 CVAE-GAN 和扩散模型更有效的课程。

想象，初始化和探索：一种有效的多智能体强化学习探索方法

Imagine, Initialize, and Explore: An Effective Exploration Method in  Multi-Agent Reinforcement Learning

Uncovering potential failure cases is a crucial step in the validation of
safety critical systems such as autonomous vehicles. Failure search may be done
through logging substantial vehicle miles in either simulation or real world
testing. Due to the sparsity of failure events, naive random search approaches
require significant amounts of vehicle operation hours to find potential system
weaknesses. As a result, adaptive searching techniques have been proposed to
efficiently explore and uncover failure trajectories of an autonomous policy in
simulation. Adaptive Stress Testing (AST) is one such method that poses the
problem of failure search as a Markov decision process and uses reinforcement
learning techniques to find high probability failures. However, this
formulation requires a probability model for the actions of all agents in the
environment. In systems where the environment actions are discrete and
dependencies among agents exist, it may be infeasible to fully characterize the
distribution or find a suitable proxy. This work proposes the use of a data
driven approach to learn a suitable classifier that tries to model how humans
identify {critical states and use this to guide failure search in AST. We show
that the incorporation of critical states into the AST framework generates
failure scenarios with increased safety violations in an autonomous driving
policy with a discrete action space.

应用数据驱动方法学习可用于指导 AST 中的故障搜索的分类器，该方法结合了重要状态，增加了一个自主驾驶策略中的安全违规案例。

使用领域专家的关键状态进行自适应故障搜索

Adaptive Failure Search Using Critical States from Domain Experts

Sampling-based motion planning techniques have emerged as an efficient
algorithmic paradigm for solving complex motion planning problems. These
approaches use a set of probing samples to construct an implicit graph
representation of the robot's state space, allowing arbitrarily accurate
representations as the number of samples increases to infinity. In practice,
however, solution trajectories only rely on a few critical states, often
defined by structure in the state space (e.g., doorways). In this work we
propose a general method to identify these critical states via graph-theoretic
techniques (betweenness centrality) and learn to predict criticality from only
local environment features. These states are then leveraged more heavily via
global connections within a hierarchical graph, termed Critical Probabilistic
Roadmaps. Critical PRMs are demonstrated to achieve up to three orders of
magnitude improvement over uniform sampling, while preserving the guarantees
and complexity of sampling-based motion planning. A video is available at
this https URL

该研究提出了一种通过图论技术和本地环境特征来识别关键状态的方法，并通过分层图中的全局连接有效利用这些关键状态来提高采样规划效率的 “Critical Probabilistic Roadmaps” 的算法，达到比均匀采样高三个数量级的效率提升。

机器人动作规划的学习式关键概率路图

Learned Critical Probabilistic Roadmaps for Robotic Motion Planning

In order to effectively interact with or supervise a robot, humans need to
have an accurate mental model of its capabilities and how it acts. Learned
neural network policies make that particularly challenging. We propose an
approach for helping end-users build a mental model of such policies. Our key
observation is that for most tasks, the essence of the policy is captured in a
few critical states: states in which it is very important to take a certain
action. Our user studies show that if the robot shows a human what its
understanding of the task's critical states is, then the human can make a more
informed decision about whether to deploy the policy, and if she does deploy
it, when she needs to take control from it at execution time.

本研究基于关键动作生成模型提出了一种新方法，通过明确展示机器人对此任务关键状态的理解，去帮助用户建立更为准确的机器人策略认知模型，从而提高机器人合作过程中人类的决策效率。