Cellular-connected unmanned aerial vehicle (UAV) is a promising technology to
unlock the full potential of UAVs in the future. However, how to achieve
ubiquitous three-dimensional (3D) communication coverage for the UAVs in the
sky is a new challenge. In this paper, we tackle this challenge by a new
coverage-aware navigation approach, which exploits the UAV's controllable
mobility to design its navigation/trajectory to avoid the cellular BSs'
coverage holes while accomplishing their missions. We formulate an UAV
trajectory optimization problem to minimize the weighted sum of its mission
completion time and expected communication outage duration, and propose a new
solution approach based on the technique of deep reinforcement learning (DRL).
To further improve the performance, we propose a new framework called
simultaneous navigation and radio mapping (SNARM), where the UAV's signal
measurement is used not only for training the deep Q network (DQN) directly,
but also to create a radio map that is able to predict the outage probabilities
at all locations in the area of interest. This thus enables the generation of
simulated UAV trajectories and predicting their expected returns, which are
then used to further train the DQN via Dyna technique, thus greatly improving
the learning efficiency.

本文提出了一种基于深度强化学习的覆盖感知导航方法，该方法利用移动无人机的可控机动性设计其导航 / 轨迹，避免了蜂窝基站覆盖漏洞，并提出了一种同时导航和无线电映射的框架。该模型通过深度 Q 网络（DQN）预测位置的中断概率，并预测期望回报，从而使得机器人学习效率大幅提升。

利用深度强化学习实现无人机的同时导航和无线电映射

Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV  with Deep Reinforcement Learning

This paper studies the path design problem for cellular-connected unmanned
aerial vehicle (UAV), which aims to minimize its mission completion time while
maintaining good connectivity with the cellular network. We first argue that
the conventional path design approach via formulating and solving optimization
problems faces several practical challenges, and then propose a new
reinforcement learning-based UAV path design algorithm by applying
\emph{temporal-difference} method to directly learn the \emph{state-value
function} of the corresponding Markov Decision Process. The proposed algorithm
is further extended by using linear function approximation with tile coding to
deal with large state space. The proposed algorithms only require the raw
measured or simulation-generated signal strength as the input and are suitable
for both online and offline implementations. Numerical results show that the
proposed path designs can successfully avoid the coverage holes of cellular
networks even in the complex urban environment.

本文研究了无人机的路径设计问题，提出了一种新的基于强化学习的算法，通过学习相应的 MDP 的状态 - 价值函数来求解，并使用瓦片编码对大状态空间进行处理。该算法使用原始测量或仿真生成的信号强度作为输入，可适用于在线和离线实现，并成功地避免了城市环境中蜂窝网络的覆盖漏洞。