The design of Wireless Networked Control System (WNCS) requires addressing
critical interactions between control and communication systems with minimal
complexity and communication overhead while providing ultra-high reliability.
This paper introduces a novel optimization theory based deep reinforcement
learning (DRL) framework for the joint design of controller and communication
systems. The objective of minimum power consumption is targeted while
satisfying the schedulability and rate constraints of the communication system
in the finite blocklength regime and stability constraint of the control
system. Decision variables include the sampling period in the control system,
and blocklength and packet error probability in the communication system. The
proposed framework contains two stages: optimization theory and DRL. In the
optimization theory stage, following the formulation of the joint optimization
problem, optimality conditions are derived to find the mathematical relations
between the optimal values of the decision variables. These relations allow the
decomposition of the problem into multiple building blocks. In the DRL stage,
the blocks that are simplified but not tractable are replaced by DRL. Via
extensive simulations, the proposed optimization theory based DRL approach is
demonstrated to outperform the optimization theory and pure DRL based
approaches, with close to optimal performance and much lower complexity.

基于优化理论的深度强化学习框架被介绍用于联合设计控制和通信系统，并针对最小化功耗目标，在满足通信系统的调度性和速率约束以及控制系统的稳定性约束的前提下，通过优化理论和深度强化学习两个阶段的组合来实现，该方法在大量的模拟实验中表现出优于优化理论和纯深度强化学习方法的性能，接近最优性能且复杂度较低。

基于优化理论的深度强化学习用于超可靠无线网络控制系统资源分配

Optimization Theory Based Deep Reinforcement Learning for Resource  Allocation in Ultra-Reliable Wireless Networked Control Systems

Wireless networked control system (WNCS) connecting sensors, controllers, and
actuators via wireless communications is a key enabling technology for highly
scalable and low-cost deployment of control systems in the Industry 4.0 era.
Despite the tight interaction of control and communications in WNCSs, most
existing works adopt separative design approaches. This is mainly because the
co-design of control-communication policies requires large and hybrid state and
action spaces, making the optimal problem mathematically intractable and
difficult to be solved effectively by classic algorithms. In this paper, we
systematically investigate deep learning (DL)-based estimator-control-scheduler
co-design for a model-unknown nonlinear WNCS over wireless fading channels. In
particular, we propose a co-design framework with the awareness of the sensor's
age-of-information (AoI) states and dynamic channel states. We propose a novel
deep reinforcement learning (DRL)-based algorithm for controller and scheduler
optimization utilizing both model-free and model-based data. An AoI-based
importance sampling algorithm that takes into account the data accuracy is
proposed for enhancing learning efficiency. We also develop novel schemes for
enhancing the stability of joint training. Extensive experiments demonstrate
that the proposed joint training algorithm can effectively solve the
estimation-control-scheduling co-design problem in various scenarios and
provide significant performance gain compared to separative design and some
benchmark policies.

探究基于无线网络控制系统的深度学习协同设计，使用新型深度强化学习算法与 AoI 技术构建控制器和调度器，提高数据准确性并增强联合训练的稳定性，以此解决传统设计方法中大量难以有效解决的问题，并在不同场景中提供显著的性能表现.