The integration of subterranean LoRaWAN and non-terrestrial networks (NTN) delivers substantial economic and societal benefits in remote agriculture and disaster rescue operations. The LoRa modulation leverages quasi-orthogonal spreading factors (SFs) to optimize data rates, airtime, coverage and energy consumption. However, it is still challenging to effectively assign SFs to end devices for minimizing co-SF interference in massive subterranean LoRaWAN NTN. To address this, we investigate a reinforcement learning (RL)-based SFs allocation scheme to optimize the system's energy efficiency (EE). To efficiently capture the device-to-environment interactions in dense networks, we proposed an SFs allocation technique using the multi-agent dueling double deep Q-network (MAD3QN) and the multi-agent advantage actor-critic (MAA2C) algorithms based on an analytical reward mechanism. Our proposed RL-based SFs allocation approach evinces better performance compared to four benchmarks in the extreme underground direct-to-satellite scenario. Remarkably, MAD3QN shows promising potentials in surpassing MAA2C in terms of convergence rate and EE.

在遥远的农业和灾害救援行动中，地下LoRaWAN与非地面网络（NTN）的整合可提供显著的经济和社会效益。然而，在大规模地下LoRaWAN NTN中有效地分配扩频因子（SFs）以最小化同样SF的干扰仍然具有挑战性。为解决这个问题，我们研究了基于强化学习（RL）的SFs分配方案，以优化系统的能源效率（EE）。在密集网络中高效捕捉设备与环境的相互作用方面，我们提出了一种使用多智能体对抗性双重深度Q网络（MAD3QN）和多智能体优势演员-评论家（MAA2C）算法的SFs分配技术，基于分析性的奖励机制。我们提出的基于强化学习的SFs分配方法在极端地下到卫星的场景中表现出比四个基准方案更好的性能。值得注意的是，MAD3QN在收敛速度和能源效率方面显示出了有希望超过MAA2C的潜力。

基于强化学习的地下LoRaWAN能效优化：直接卫星接入场景