This paper considers the challenging tasks of Multi-Agent Reinforcement
Learning (MARL) under partial observability, where each agent only sees her own
individual observations and actions that reveal incomplete information about
the underlying state of system. This paper studies these tasks under the
general model of multiplayer general-sum Partially Observable Markov Games
(POMGs), which is significantly larger than the standard model of Imperfect
Information Extensive-Form Games (IIEFGs). We identify a rich subclass of POMGs
-- weakly revealing POMGs -- in which sample-efficient learning is tractable.
In the self-play setting, we prove that a simple algorithm combining optimism
and Maximum Likelihood Estimation (MLE) is sufficient to find approximate Nash
equilibria, correlated equilibria, as well as coarse correlated equilibria of
weakly revealing POMGs, in a polynomial number of samples when the number of
agents is small. In the setting of playing against adversarial opponents, we
show that a variant of our optimistic MLE algorithm is capable of achieving
sublinear regret when being compared against the optimal maximin policies. To
our best knowledge, this work provides the first line of sample-efficient
results for learning POMGs.

本文研究了多智能体强化学习在部分可观察性下的挑战性任务，其中每个智能体只能看到自己的观察和动作。我们通过考虑广义模型的部分可观察马尔科夫博弈，证明了一个富裕的子类可以使用样本高效的学习方法，从而找到弱显式部分可观察马尔科夫博弈的近似纳什均衡、相关均衡以及粗略相关均衡，当代理数量很小时可在多项式样本复杂度内学得。

部分可观马尔可夫博弈中高效学习的样本有效强化学习

Sample-Efficient Reinforcement Learning of Partially Observable Markov Games

The capability to learn and adapt to changes in the driving environment is
crucial for developing autonomous driving systems that are scalable beyond
geo-fenced operational design domains. Deep Reinforcement Learning (RL)
provides a promising and scalable framework for developing adaptive learning
based solutions. Deep RL methods usually model the problem as a (Partially
Observable) Markov Decision Process in which an agent acts in a stationary
environment to learn an optimal behavior policy. However, driving involves
complex interaction between multiple, intelligent (artificial or human) agents
in a highly non-stationary environment. In this paper, we propose the use of
Partially Observable Markov Games(POSG) for formulating the connected
autonomous driving problems with realistic assumptions. We provide a taxonomy
of multi-agent learning environments based on the nature of tasks, nature of
agents and the nature of the environment to help in categorizing various
autonomous driving problems that can be addressed under the proposed
formulation. As our main contributions, we provide MACAD-Gym, a Multi-Agent
Connected, Autonomous Driving agent learning platform for furthering research
in this direction. Our MACAD-Gym platform provides an extensible set of
Connected Autonomous Driving (CAD) simulation environments that enable the
research and development of Deep RL- based integrated sensing, perception,
planning and control algorithms for CAD systems with unlimited operational
design domain under realistic, multi-agent settings. We also share the
MACAD-Agents that were trained successfully using the MACAD-Gym platform to
learn control policies for multiple vehicle agents in a partially observable,
stop-sign controlled, 3-way urban intersection environment with raw (camera)
sensor observations.

该论文提出了使用部分可观测马尔可夫博弈来构建具有现实情况的无人驾驶问题，并提供了 MACAD-Gym 平台来进一步研究和开发基于深度强化学习的集成感知、计划和控制算法。