Recent works have validated the possibility of improving energy efficiency in
radio access networks (RANs), achieved by dynamically turning on/off some base
stations (BSs). In this paper, we extend the research over BS switching
operations, which should match up with traffic load variations. Instead of
depending on the dynamic traffic loads which are still quite challenging to
precisely forecast, we firstly formulate the traffic variations as a Markov
decision process. Afterwards, in order to foresightedly minimize the energy
consumption of RANs, we design a reinforcement learning framework based BS
switching operation scheme. Furthermore, to avoid the underlying curse of
dimensionality in reinforcement learning, a transfer actor-critic algorithm
(TACT), which utilizes the transferred learning expertise in historical periods
or neighboring regions, is proposed and provably converges. In the end, we
evaluate our proposed scheme by extensive simulations under various practical
configurations and show that the proposed TACT algorithm contributes to a
performance jumpstart and demonstrates the feasibility of significant energy
efficiency improvement at the expense of tolerable delay performance.

本文研究了如何通过基站开关操作来降低无线电接入网络的能量消耗。为此，我们根据马尔科夫决策过程对流量变化进行建模，并设计了一个基于强化学习的开关操作方案，其中利用了历史周期或邻近区域中的传递学习专业知识的转移演员 - 评论家算法。通过在各种实际配置下的广泛模拟评估，我们证明了所提出的算法可以显著提高能源效率，而延迟性能的牺牲是可以接受的。