In this work, we build upon the offline reinforcement learning algorithm TD7, which incorporates State-Action Learned Embeddings (SALE) and LAP, and propose a model-free actor-critic algorithm that integrates ensemble Q-networks and a gradient diversity penalty from EDAC. The ensemble Q-networks effectively address the challenge of out-of-distribution actions by introducing penalties that guide the actor network to focus on in-distribution actions. Meanwhile, the gradient diversity penalty encourages diverse Q-value gradients, further suppressing overestimation for out-of-distribution actions. Additionally, our method retains an adjustable behavior cloning (BC) term that directs the actor network toward dataset actions during early training stages, while gradually reducing its influence as the precision of the Q-ensemble improves. These enhancements work synergistically to improve training stability and accuracy. Experimental results on the D4RL MuJoCo benchmarks demonstrate that our algorithm achieves superior convergence speed, stability, and performance compared to existing methods.

本研究解决了离线强化学习中处理分布外动作的挑战，通过提出一种集成Q网络的无模型演员-评论家算法，增强了训练的稳定性和准确性。研究的关键在于引入梯度多样性惩罚和可调行为克隆项，有效抑制了分布外动作的估计过高现象，并逐步优化演员网络的表现。实验结果表明，该算法在D4RL MuJoCo基准上具有更快的收敛速度和更优的性能。

基于SALE的离线强化学习与集成Q网络