This article addresses the problem of Ultra Reliable Low Latency
Communications (URLLC) in wireless networks, a framework with particularly
stringent constraints imposed by many Internet of Things (IoT) applications
from diverse sectors. We propose a novel Deep Reinforcement Learning (DRL)
scheduling algorithm, named NOMA-PPO, to solve the Non-Orthogonal Multiple
Access (NOMA) uplink URLLC scheduling problem involving strict deadlines. The
challenge of addressing uplink URLLC requirements in NOMA systems is related to
the combinatorial complexity of the action space due to the possibility to
schedule multiple devices, and to the partial observability constraint that we
impose to our algorithm in order to meet the IoT communication constraints and
be scalable. Our approach involves 1) formulating the NOMA-URLLC problem as a
Partially Observable Markov Decision Process (POMDP) and the introduction of an
agent state, serving as a sufficient statistic of past observations and
actions, enabling a transformation of the POMDP into a Markov Decision Process
(MDP); 2) adapting the Proximal Policy Optimization (PPO) algorithm to handle
the combinatorial action space; 3) incorporating prior knowledge into the
learning agent with the introduction of a Bayesian policy. Numerical results
reveal that not only does our approach outperform traditional multiple access
protocols and DRL benchmarks on 3GPP scenarios, but also proves to be robust
under various channel and traffic configurations, efficiently exploiting
inherent time correlations.

通过提出一种名为 NOMA-PPO 的新型深度强化学习调度算法，本文解决了无线网络中超可靠低延迟通信（URLLC）的问题，也就是在物联网（IoT）应用中施加的严格约束。该方法通过将 NOMA-URLLC 问题转化为部分可观察马尔可夫决策过程（POMDP）并引入一种代理状态，成功将 POMDP 转化为马尔可夫决策过程（MDP），并且适应了组合动作空间，同时还利用贝叶斯策略技术融合了先前的知识，结果表明该方法不仅在 3GPP 场景中优于传统的多路访问协议和 DRL 基准，而且在各种信道和流量配置下都能有效地利用时间相关性，显示出鲁棒的性能。