Remaining competitive in future conflicts with technologically-advanced
competitors requires us to accelerate our research and development in
artificial intelligence (AI) for wargaming. More importantly, leveraging
machine learning for intelligent combat behavior development will be key to one
day achieving superhuman performance in this domain--elevating the quality and
accelerating the speed of our decisions in future wars. Although deep
reinforcement learning (RL) continues to show promising results in intelligent
agent behavior development in games, it has yet to perform at or above the
human level in the long-horizon, complex tasks typically found in combat
modeling and simulation. Capitalizing on the proven potential of RL and recent
successes of hierarchical reinforcement learning (HRL), our research is
investigating and extending the use of HRL to create intelligent agents capable
of performing effectively in these large and complex simulation environments.
Our ultimate goal is to develop an agent capable of superhuman performance that
could then serve as an AI advisor to military planners and decision-makers.
This papers covers our ongoing approach and the first three of our five
research areas aimed at managing the exponential growth of computations that
have thus far limited the use of AI in combat simulations: (1) developing an
HRL training framework and agent architecture for combat units; (2) developing
a multi-model framework for agent decision-making; (3) developing
dimension-invariant observation abstractions of the state space to manage the
exponential growth of computations; (4) developing an intrinsic rewards engine
to enable long-term planning; and (5) implementing this framework into a
higher-fidelity combat simulation.

未来与先进技术竞争对手保持竞争力需要加快我们在战争推演中人工智能（AI）的研发。更重要的是，利用机器学习进行智能战斗行为开发将成为未来实现超人类表现的关键，提高未来战争决策的质量和加速速度。尽管深度强化学习（RL）在智能代理行为开发方面仍然显示出有希望的结果，但在战斗模拟中通常遇到的长远、复杂任务中尚未达到或超过人类水平。充分利用 RL 的成功潜力和分层强化学习（HRL）的最新成功，我们的研究正在探索和扩展 HRL 的使用，以创建能在这些庞大而复杂的模拟环境中有效执行的智能代理。我们的最终目标是开发出一个能够具有超人类表现的代理，然后作为军事规划者和决策者的人工智能顾问。本文介绍了我们正在进行的方法以及我们的五个研究领域中的前三个，旨在管理迅速增长的计算，这些计算迄今为止限制了 AI 在战斗模拟中的使用: （1）为作战单位开发 HRL 训练框架和代理架构；（2）开发代理决策的多模型框架；（3）开发维度不变的状态空间观察抽象化以管理计算的指数增长；（4）开发内在奖励引擎以实现长期规划；（5）将该框架实施到更高保真度的战斗模拟中。