Despite the recent advancements in offline RL, no unified algorithm could
achieve superior performance across a broad range of tasks. Offline
\textit{value function learning}, in particular, struggles with sparse-reward,
long-horizon tasks due to the difficulty of solving credit assignment and
extrapolation errors that accumulates as the horizon of the task grows.~On the
other hand, models that can perform well in long-horizon tasks are designed
specifically for goal-conditioned tasks, which commonly perform worse than
value function learning methods on short-horizon, dense-reward scenarios. To
bridge this gap, we propose a hierarchical planner designed for offline RL
called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level,
named D-Conductor, which guides the low-level policy through sub-goals. At the
low level, we used a Q-learning based approach called the Q-Performer to
accomplish these sub-goals. Our experimental results suggest that PlanDQ can
achieve superior or competitive performance on D4RL continuous control
benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.

提出一种用于离线强化学习的分层规划器 PlanDQ，其中包括高层的扩散式规划器 D-Conductor，用于指导低层策略实现子目标，并采用 Q-learning 的方法 Q-Performer 来完成这些子目标，实验结果表明 PlanDQ 在 D4RL 连续控制基准任务以及 AntMaze、Kitchen 和 Calvin 等长时间跨度任务上实现了优越或有竞争力的性能。

PlanDQ: 分层计划编排通过 D-Conductor 和 Q-Performer

PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer

We investigate the problem of decentralized multi-agent navigation tasks,
where multiple agents need to reach initially unassigned targets in a limited
time. Classical planning-based methods suffer from expensive computation
overhead at each step and offer limited expressiveness for complex cooperation
strategies. In contrast, reinforcement learning (RL) has recently become a
popular paradigm for addressing this issue. However, RL struggles with low data
efficiency and cooperation when directly exploring (nearly) optimal policies in
the large search space, especially with an increased agent number (e.g., 10+
agents) or in complex environments (e.g., 3D simulators). In this paper, we
propose Multi-Agent Scalable GNN-based P lanner (MASP), a goal-conditioned
hierarchical planner for navigation tasks with a substantial number of agents.
MASP adopts a hierarchical framework to divide a large search space into
multiple smaller spaces, thereby reducing the space complexity and accelerating
training convergence. We also leverage graph neural networks (GNN) to model the
interaction between agents and goals, improving goal achievement. Besides, to
enhance generalization capabilities in scenarios with unseen team sizes, we
divide agents into multiple groups, each with a previously trained number of
agents. The results demonstrate that MASP outperforms classical planning-based
competitors and RL baselines, achieving a nearly 100% success rate with minimal
training data in both multi-agent particle environments (MPE) with 50 agents
and a quadrotor 3-dimensional environment (OmniDrones) with 20 agents.
Furthermore, the learned policy showcases zero-shot generalization across
unseen team sizes.

我们提出了基于多智能体可扩展图神经网络的分层规划器 (MASP)，用于具有大量智能体的导航任务，通过将搜索空间分成多个较小的空间以减少复杂度和加快训练收敛速度，并利用图神经网络建模智能体和目标之间的相互作用，提高目标实现能力，并在未见过的团队规模工作场景中展示了零 - shot 泛化能力。

MASP: 面向多智能体导航的可扩展基于 GNN 的规划

MASP: Scalable GNN-based Planning for Multi-Agent Navigation

A typical way in which a machine acquires knowledge from humans is by
programming. Compared to learning from demonstrations or experiences,
programmatic learning allows the machine to acquire a novel skill as soon as
the program is written, and, by building a library of programs, a machine can
quickly learn how to perform complex tasks. However, as programs often take
their execution contexts for granted, they are brittle when the contexts
change, making it difficult to adapt complex programs to new contexts. We
present natural programming, a library learning system that combines
programmatic learning with a hierarchical planner. Natural programming
maintains a library of decompositions, consisting of a goal, a linguistic
description of how this goal decompose into sub-goals, and a concrete instance
of its decomposition into sub-goals. A user teaches the system via curriculum
building, by identifying a challenging yet not impossible goal along with
linguistic hints on how this goal may be decomposed into sub-goals. The system
solves for the goal via hierarchical planning, using the linguistic hints to
guide its probability distribution in proposing the right plans. The system
learns from this interaction by adding newly found decompositions in the
successful search into its library. Simulated studies and a human experiment
(n=360) on a controlled environment demonstrate that natural programming can
robustly compose programs learned from different users and contexts, adapting
faster and solving more complex tasks when compared to programmatic baselines.

机器学习中的自然编程通过结合基于程序的学习和分层规划，通过使用语言提示来指导规划提案的概率分布，从而实现了从不同用户和环境中学习并快速适应和解决更复杂任务的目标分解与复杂任务的解决。