Dynamic radio resource management (RRM) in wireless networks presents
significant challenges, particularly in the context of Radio Access Network
(RAN) slicing. This technology, crucial for catering to varying user
requirements, often grapples with complex optimization scenarios. Existing
Reinforcement Learning (RL) approaches, while achieving good performance in RAN
slicing, typically rely on online algorithms or behavior cloning. These methods
necessitate either continuous environmental interactions or access to
high-quality datasets, hindering their practical deployment. Towards addressing
these limitations, this paper introduces offline RL to solving the RAN slicing
problem, marking a significant shift towards more feasible and adaptive RRM
methods. We demonstrate how offline RL can effectively learn near-optimal
policies from sub-optimal datasets, a notable advancement over existing
practices. Our research highlights the inherent flexibility of offline RL,
showcasing its ability to adjust policy criteria without the need for
additional environmental interactions. Furthermore, we present empirical
evidence of the efficacy of offline RL in adapting to various service-level
requirements, illustrating its potential in diverse RAN slicing scenarios.

该研究介绍了离线强化学习（RL）在解决无线网络中的无线电资源管理（RRM）问题方面的应用，展示了它学习接近最优策略的能力，适应不同的服务级别要求，并在各种 RAN 分割场景中发挥潜力。

离线强化学习推进 RAN 切片技术

Advancing RAN Slicing with Offline Reinforcement Learning

Deep reinforcement learning (DRL) algorithms have recently gained wide
attention in the wireless networks domain. They are considered promising
approaches for solving dynamic radio resource management (RRM) problems in
next-generation networks. Given their capabilities to build an approximate and
continuously updated model of the wireless network environments, DRL algorithms
can deal with the multifaceted complexity of such environments. Nevertheless,
several challenges hinder the practical adoption of DRL in commercial networks.
In this article, we first discuss two key practical challenges that are faced
but rarely tackled when developing DRL-based RRM solutions. We argue that it is
inevitable to address these DRL-related challenges for DRL to find its way to
RRM commercial solutions. In particular, we discuss the need to have safe and
accelerated DRL-based RRM solutions that mitigate the slow convergence and
performance instability exhibited by DRL algorithms. We then review and
categorize the main approaches used in the RRM domain to develop safe and
accelerated DRL-based solutions. Finally, a case study is conducted to
demonstrate the importance of having safe and accelerated DRL-based RRM
solutions. We employ multiple variants of transfer learning (TL) techniques to
accelerate the convergence of intelligent radio access network (RAN) slicing
DRL-based controllers. We also propose a hybrid TL-based approach and sigmoid
function-based rewards as examples of safe exploration in DRL-based RAN
slicing.

本文主要讨论了 DRL 技术在无线网络领域中的应用，提出了安全性和加速性等难点问题，并对安全加速 DRL 的解决方案进行了分类和讨论，最后通过案例研究展示了如何通过传输学习和安全探索来加速智能无线接入网络的 DRL 控制器收敛。