In the multiple unmanned aerial vehicle (UAV)- assisted downlink
communication, it is challenging for UAV base stations (UAV BSs) to realize
trajectory design and resource assignment in unknown environments. The
cooperation and competition between UAV BSs in the communication network leads
to a Markov game problem. Multi-agent reinforcement learning is a significant
solution for the above decision-making. However, there are still many common
issues, such as the instability of the system and low utilization of historical
data, that limit its application. In this paper, a novel graph-attention
multi-agent trust region (GA-MATR) reinforcement learning framework is proposed
to solve the multi-UAV assisted communication problem. Graph recurrent network
is introduced to process and analyze complex topology of the communication
network, so as to extract useful information and patterns from observational
information. The attention mechanism provides additional weighting for conveyed
information, so that the critic network can accurately evaluate the value of
behavior for UAV BSs. This provides more reliable feedback signals and helps
the actor network update the strategy more effectively. Ablation simulations
indicate that the proposed approach attains improved convergence over the
baselines. UAV BSs learn the optimal communication strategies to achieve their
maximum cumulative rewards. Additionally, multi-agent trust region method with
monotonic convergence provides an estimated Nash equilibrium for the multi-UAV
assisted communication Markov game.

该论文提出了一种新的图注意多智能体信任域（GA-MATR）强化学习框架，用于解决多无人机辅助通信问题，通过引入图递归网络处理和分析通信网络的复杂拓扑结构，从观测信息中提取有用的信息和模式，提供额外的权重，并利用批评网络准确评估无人机基站行为的价值，以实现更可靠的反馈信号，并帮助演员网络更有效地更新策略。模拟实验表明，该方法具有优于基准线的收敛性能，无人机基站学习到了达到最大累积奖励的最佳通信策略，同时，多智能体信任域方法具有单调收敛性，为多无人机辅助通信马尔可夫博弈提供了估计的纳什均衡点。