Unmanned Aerial Vehicles (UAVs) are increasingly used as aerial base stations
to provide ad hoc communications infrastructure. Building upon prior research
efforts which consider either static nodes, 2D trajectories or single UAV
systems, this paper focuses on the use of multiple UAVs for providing wireless
communication to mobile users in the absence of terrestrial communications
infrastructure. In particular, we jointly optimize UAV 3D trajectory and NOMA
power allocation to maximize system throughput. Firstly, a weighted
K-means-based clustering algorithm establishes UAV-user associations at regular
intervals. The efficacy of training a novel Shared Deep Q-Network (SDQN) with
action masking is then explored. Unlike training each UAV separately using DQN,
the SDQN reduces training time by using the experiences of multiple UAVs
instead of a single agent. We also show that SDQN can be used to train a
multi-agent system with differing action spaces. Simulation results confirm
that: 1) training a shared DQN outperforms a conventional DQN in terms of
maximum system throughput (+20%) and training time (-10%); 2) it can converge
for agents with different action spaces, yielding a 9% increase in throughput
compared to mutual learning algorithms; and 3) combining NOMA with an SDQN
architecture enables the network to achieve a better sum rate compared with
existing baseline schemes.

本篇论文研究使用多个无人机作为基站，通过联合优化无人机三维轨迹和 NOMA 功率分配来最大化系统吞吐量，并使用基于加权 K-means 的聚类算法建立无人机 - 用户关联，研究表明使用共享的 Deep Q-Network 在系统吞吐量和训练时间上表现优于传统的 DQN，同时还可以收敛于具有不同动作空间的多个智能体，与 NOMA 的结合可以实现比现有基线方案更好的总速率。