Risk-sensitive linear quadratic regulator is one of the most fundamental
problems in risk-sensitive optimal control. In this paper, we study online
adaptive control of risk-sensitive linear quadratic regulator in the finite
horizon episodic setting. We propose a simple least-squares greedy algorithm
and show that it achieves $\widetilde{\mathcal{O}}(\log N)$ regret under a
specific identifiability assumption, where $N$ is the total number of episodes.
If the identifiability assumption is not satisfied, we propose incorporating
exploration noise into the least-squares-based algorithm, resulting in an
algorithm with $\widetilde{\mathcal{O}}(\sqrt{N})$ regret. To our best
knowledge, this is the first set of regret bounds for episodic risk-sensitive
linear quadratic regulator. Our proof relies on perturbation analysis of
less-standard Riccati equations for risk-sensitive linear quadratic control,
and a delicate analysis of the loss in the risk-sensitive performance criterion
due to applying the suboptimal controller in the online learning process.

在线自适应控制有限时域周期性设置下风险敏感线性二次调节器的回避范围研究及拟合泛化条件的简化最小二乘贪婪算法

一种基于风险敏感的时序性线性二次调节器的遗憾界限

Regret Bounds for Episodic Risk-Sensitive Linear Quadratic Regulator

We consider the problem of online adaptive control of the linear quadratic
regulator, where the true system parameters are unknown. We prove new upper and
lower bounds demonstrating that the optimal regret scales as
$\widetilde{\Theta}({\sqrt{d_{\mathbf{u}}^2 d_{\mathbf{x}} T}})$, where $T$ is
the number of time steps, $d_{\mathbf{u}}$ is the dimension of the input space,
and $d_{\mathbf{x}}$ is the dimension of the system state. Notably, our lower
bounds rule out the possibility of a $\mathrm{poly}(\log{}T)$-regret algorithm,
which had been conjectured due to the apparent strong convexity of the problem.
Our upper bound is attained by a simple variant of $\textit{{certainty
equivalent control}}$, where the learner selects control inputs according to
the optimal controller for their estimate of the system while injecting
exploratory random noise. While this approach was shown to achieve
$\sqrt{T}$-regret by (Mania et al. 2019), we show that if the learner
continually refines their estimates of the system matrices, the method attains
optimal dimension dependence as well.
Central to our upper and lower bounds is a new approach for controlling
perturbations of Riccati equations called the $\textit{self-bounding ODE
method}$, which we use to derive suboptimality bounds for the certainty
equivalent controller synthesized from estimated system dynamics. This in turn
enables regret upper bounds which hold for $\textit{any stabilizable instance}$
and scale with natural control-theoretic quantities.

在处理未知真实系统参数的在线自适应控制问题中，使用新的上下界结论证明误差的最优性跟时间步数，输入空间和系统状态空间的维度呈现为～(T*d_u^2*d_x)^1/2, 并引入自绑定 ODE 方法控制 Riccati 方程扰动，从而实现任意可控系统实例的回归上界。同时，提出对估计的系统动力学进行合成的确定性等效控制器。