This work presents a sustainable multi-agent deep reinforcement learning
framework capable of selectively scaling parallelized training workloads
on-demand, and transferring the trained policies from simulation to reality
using minimal hardware resources. We introduce AutoDRIVE Ecosystem as an
enabling digital twin framework to train, deploy, and transfer cooperative as
well as competitive multi-agent reinforcement learning policies from simulation
to reality. Particularly, we first investigate an intersection traversal
problem of 4 cooperative vehicles (Nigel) that share limited state information
in single as well as multi-agent learning settings using a common policy
approach. We then investigate an adversarial autonomous racing problem of 2
vehicles (F1TENTH) using an individual policy approach. In either set of
experiments, a decentralized learning architecture was adopted, which allowed
robust training and testing of the policies in stochastic environments. The
agents were provided with realistically sparse observation spaces, and were
restricted to sample control actions that implicitly satisfied the imposed
kinodynamic and safety constraints. The experimental results for both problem
statements are reported in terms of quantitative metrics and qualitative
remarks for training as well as deployment phases. We also discuss agent and
environment parallelization techniques adopted to efficiently accelerate MARL
training, while analyzing their computational performance. Finally, we
demonstrate a resource-aware transition of the trained policies from simulation
to reality using the proposed digital twin framework.

本文提出了一种可持续的多智能体深度强化学习框架，能够按需选择性地扩展并行训练工作负载，并利用最小的硬件资源将训练策略从模拟转移到真实环境中。我们引入了 AutoDRIVE 生态系统作为一种使合作和竞争的多智能体强化学习策略能够从模拟环境转移到现实环境中进行训练、部署和传输的数字孪生框架。在研究中，我们首先探讨了 4 个合作车辆（Nigel）在单个智能体和多智能体学习环境中共享有限状态信息时的交叉路口问题，并采用了共同的策略方法。然后，我们使用个体策略方法研究了 2 辆车（F1TENTH）的对抗自主赛车问题。在实验中，我们采用了分散式学习架构，能够在不确定性环境中进行稳健的策略训练和测试。智能体只能观察到有限的观测空间，并且只能采样满足运动动力学和安全约束的控制动作。我们以定量指标和定性评述的方式报告了两个问题陈述的实验结果，包括训练和部署阶段。我们还讨论了用于加速多智能体强化学习训练的智能体和环境并行化技术，并分析了它们的计算性能。最后，我们使用提出的数字孪生框架展示了从模拟环境到真实环境中训练策略的资源感知过渡。