In Part I of this two-part paper (Multi-Timescale Control and Communications
with Deep Reinforcement Learning -- Part I: Communication-Aware Vehicle
Control), we decomposed the multi-timescale control and communications (MTCC)
problem in Cellular Vehicle-to-Everything (C-V2X) system into a
communication-aware Deep Reinforcement Learning (DRL)-based platoon control
(PC) sub-problem and a control-aware DRL-based radio resource allocation (RRA)
sub-problem. We focused on the PC sub-problem and proposed the MTCC-PC
algorithm to learn an optimal PC policy given an RRA policy. In this paper
(Part II), we first focus on the RRA sub-problem in MTCC assuming a PC policy
is given, and propose the MTCC-RRA algorithm to learn the RRA policy.
Specifically, we incorporate the PC advantage function in the RRA reward
function, which quantifies the amount of PC performance degradation caused by
observation delay. Moreover, we augment the state space of RRA with PC action
history for a more well-informed RRA policy. In addition, we utilize reward
shaping and reward backpropagation prioritized experience replay (RBPER)
techniques to efficiently tackle the multi-agent and sparse reward problems,
respectively. Finally, a sample- and computational-efficient training approach
is proposed to jointly learn the PC and RRA policies in an iterative process.
In order to verify the effectiveness of the proposed MTCC algorithm, we
performed experiments using real driving data for the leading vehicle, where
the performance of MTCC is compared with those of the baseline DRL algorithms.

我们将多时间尺度控制和通信 (MTCC) 问题分解为基于深度强化学习 (DRL) 的车队控制 (PC) 子问题和基于 DRL 的无线资源分配 (RRA) 子问题，并提出了用于学习最优 PC 策略的 MTCC-PC 算法和用于学习 RRA 策略的 MTCC-RRA 算法。我们采用奖励塑形和奖励反向传播优先经验回放 (RBPER) 技巧来高效地解决多智能体和稀疏奖励问题，并提出了一种样本和计算高效的训练方法来共同学习 PC 和 RRA 策略。通过使用真实驾驶数据进行实验，将 MTCC 的性能与基准 DRL 算法进行了比较，验证了所提出的 MTCC 算法的有效性。

深度强化学习的多时间尺度控制和通信 - 第二部分：控制感知的无线资源分配

Multi-Timescale Control and Communications with Deep Reinforcement  Learning -- Part II: Control-Aware Radio Resource Allocation

The advent of the wireless communications systems augurs new cutting-edge
technologies, including self-driving vehicles, unmanned aerial systems,
autonomous robots, Internet-of-Things, and virtual reality. These technologies
require high data rates, ultra-low latency, and high reliability, all of which
are promised by the fifth generation of wireless communication systems (5G).
Many research groups state that 5G cannot meet its demands without artificial
intelligence (AI) integration as 5G wireless networks are expected to generate
unprecedented traffic giving wireless research designers access to big data
that can help in predicting the demands and adjust cell designs to meet the
users requirements. Subsequently, many researchers applied AI in many aspects
of 5G wireless communication design including radio resource allocation,
network management, and cyber-security. In this paper, we provide an in-depth
review of AI for 5G wireless communication systems. In this respect, the aim of
this paper is to survey AI in 5G wireless communication systems by discussing
many case studies and the associated challenges, and shedding new light on
future research directions for leveraging AI in 5G wireless communications.

本文评估了基于人工智能的 5G 无线通信系统，包括网络管理、无线资源分配和网络安全等。

5G 无线系统中的人工智能：机遇、挑战和未来研究方向

Artificial Intelligence for 5G Wireless Systems: Opportunities,  Challenges, and Future Research Directions

In this article, we study a Radio Resource Allocation (RRA) that was
formulated as a non-convex optimization problem whose main aim is to maximize
the spectral efficiency subject to satisfaction guarantees in multiservice
wireless systems. This problem has already been previously investigated in the
literature and efficient heuristics have been proposed. However, in order to
assess the performance of Machine Learning (ML) algorithms when solving
optimization problems in the context of RRA, we revisit that problem and
propose a solution based on a Reinforcement Learning (RL) framework.
Specifically, a distributed optimization method based on multi-agent deep RL is
developed, where each agent makes its decisions to find a policy by interacting
with the local environment, until reaching convergence. Thus, this article
focuses on an application of RL and our main proposal consists in a new deep RL
based approach to jointly deal with RRA, satisfaction guarantees and Quality of
Service (QoS) constraints in multiservice celular networks. Lastly, through
computational simulations we compare the state-of-art solutions of the
literature with our proposal and we show a near optimal performance of the
latter in terms of throughput and outage rate.

本文旨在研究一种利用强化学习算法在无线多业务系统中处理无线电资源分配问题的方法，通过在本地环境中与局部互动来寻找策略，实现了关于 9LTE 系统满足保证和 QoS 约束的无线资源分配。此外，通过计算模拟，本文与文献中的最新解决方案进行比较，并展示了后者在吞吐量和故障率方面近乎最优的性能。

多业务网络中基于 QoS 限制的资源分配的深度强化学习

Deep Reinforcement Learning for QoS-Constrained Resource Allocation in  Multiservice Networks

NarrowBand-Internet of Things (NB-IoT) is an emerging cellular-based
technology that offers a range of flexible configurations for massive IoT radio
access from groups of devices with heterogeneous requirements. A configuration
specifies the amount of radio resource allocated to each group of devices for
random access and for data transmission. Assuming no knowledge of the traffic
statistics, there exists an important challenge in "how to determine the
configuration that maximizes the long-term average number of served IoT devices
at each Transmission Time Interval (TTI) in an online fashion". Given the
complexity of searching for optimal configuration, we first develop real-time
configuration selection based on the tabular Q-learning (tabular-Q), the Linear
Approximation based Q-learning (LA-Q), and the Deep Neural Network based
Q-learning (DQN) in the single-parameter single-group scenario. Our results
show that the proposed reinforcement learning based approaches considerably
outperform the conventional heuristic approaches based on load estimation
(LE-URC) in terms of the number of served IoT devices. This result also
indicates that LA-Q and DQN can be good alternatives for tabular-Q to achieve
almost the same performance with much less training time. We further advance
LA-Q and DQN via Actions Aggregation (AA-LA-Q and AA-DQN) and via Cooperative
Multi-Agent learning (CMA-DQN) for the multi-parameter multi-group scenario,
thereby solve the problem that Q-learning agents do not converge in
high-dimensional configurations. In this scenario, the superiority of the
proposed Q-learning approaches over the conventional LE-URC approach
significantly improves with the increase of configuration dimensions, and the
CMA-DQN approach outperforms the other approaches in both throughput and
training efficiency.

本文阐述了如何用智能算法解决 NB-IoT 中设备组的无线资源分配问题，并提出了基于 Q-Learning 算法的智能化方案，这个方案在吞吐量和训练效率方面显著优于传统的方案。