This paper proposes a novel design on the wireless powered communication
network (WPCN) in dynamic environments under the assistance of multiple
unmanned aerial vehicles (UAVs). Unlike the existing studies, where the
low-power wireless nodes (WNs) often conform to the coherent
harvest-then-transmit protocol, under our newly proposed double-threshold based
WN type updating rule, each WN can dynamically and repeatedly update its WN
type as an E-node for non-linear energy harvesting over time slots or an I-node
for transmitting data over sub-slots. To maximize the total transmission data
size of all the WNs over T slots, each of the UAVs individually determines its
trajectory and binary wireless energy transmission (WET) decisions over times
slots and its binary wireless data collection (WDC) decisions over sub-slots,
under the constraints of each UAV's limited on-board energy and each WN's node
type updating rule. However, due to the UAVs' tightly-coupled trajectories with
their WET and WDC decisions, as well as each WN's time-varying battery energy,
this problem is difficult to solve optimally. We then propose a new multi-agent
based hierarchical deep reinforcement learning (MAHDRL) framework with two
tiers to solve the problem efficiently, where the soft actor critic (SAC)
policy is designed in tier-1 to determine each UAV's continuous trajectory and
binary WET decision over time slots, and the deep-Q learning (DQN) policy is
designed in tier-2 to determine each UAV's binary WDC decisions over sub-slots
under the given UAV trajectory from tier-1. Both of the SAC policy and the DQN
policy are executed distributively at each UAV. Finally, extensive simulation
results are provided to validate the outweighed performance of the proposed
MAHDRL approach over various state-of-the-art benchmarks.

该研究提出了一种在多个无人机的辅助下，针对动态环境中的无线供能通信网络（WPCN）的新设计。通过引入新的基于双阈值的无线节点类型更新规则，以及多智能体深度强化学习（MAHDRL）框架，能有效解决航迹决策和能源传输与数据收集决策的优化问题。通过大量的仿真实验结果验证了该方法在各种国际先进基准测试中的卓越性能。