Real-world decision-making tasks are usually partially observable Markov
decision processes (POMDPs), where the state is not fully observable. Recent
progress has demonstrated that recurrent reinforcement learning (RL), which
consists of a context encoder based on recurrent neural networks (RNNs) for
unobservable state prediction and a multilayer perceptron (MLP) policy for
decision making, can mitigate partial observability and serve as a robust
baseline for POMDP tasks. However, previous recurrent RL methods face training
stability issues due to the gradient instability of RNNs. In this paper, we
propose Recurrent Off-policy RL with Context-Encoder-Specific Learning Rate
(RESeL) to tackle this issue. Specifically, RESeL uses a lower learning rate
for context encoder than other MLP layers to ensure the stability of the former
while maintaining the training efficiency of the latter. We integrate this
technique into existing off-policy RL methods, resulting in the RESeL
algorithm. We evaluated RESeL in 18 POMDP tasks, including classic, meta-RL,
and credit assignment scenarios, as well as five MDP locomotion tasks. The
experiments demonstrate significant improvements in training stability with
RESeL. Comparative results show that RESeL achieves notable performance
improvements over previous recurrent RL baselines in POMDP tasks, and is
competitive with or even surpasses state-of-the-art methods in MDP tasks.
Further ablation studies highlight the necessity of applying a distinct
learning rate for the context encoder.

使用一种称为 RESeL 的方法改善了循环强化学习（RL）中的训练稳定性问题，并在部分可观察的决策任务中获得了显著的性能改进。

高效的递归离策略强化学习需要一个上下文编码器特定的学习率

Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific  Learning Rate

Model Predictive Control (MPC) is attracting tremendous attention in the
autonomous driving task as a powerful control technique. The success of an MPC
controller strongly depends on an accurate internal dynamics model. However,
the static parameters, usually learned by system identification, often fail to
adapt to both internal and external perturbations in real-world scenarios. In
this paper, we firstly (1) reformulate the problem as a Partially Observed
Markov Decision Process (POMDP) that absorbs the uncertainties into
observations and maintains Markov property into hidden states; and (2) learn a
recurrent policy continually adapting the parameters of the dynamics model via
Recurrent Reinforcement Learning (RRL) for optimal and adaptive control; and
(3) finally evaluate the proposed algorithm (referred as $\textit{MPC-RRL}$) in
CARLA simulator and leading to robust behaviours under a wide range of
perturbations.

通过解决 MPC 控制器在现实场景下系统识别学习失败的问题，将其转化为部分观察马尔科夫决策过程，通过循环强化学习不断地适应动态模型参数，该论文提出了一种自适应控制算法 (MPC-RRL)，最终在 CARLA 模拟器中得出了具有鲁棒性和可靠性的自动驾驶控制效果。