Deep reinforcement learning (DRL) methods such as the Deep Q-Network (DQN)
have achieved state-of-the-art results in a variety of challenging,
high-dimensional domains. This success is mainly attributed to the power of
deep neural networks to learn rich domain representations for approximating the
value function or policy. Batch reinforcement learning methods with linear
representations, on the other hand, are more stable and require less hyper
parameter tuning. Yet, substantial feature engineering is necessary to achieve
good results. In this work we propose a hybrid approach -- the Least Squares
Deep Q-Network (LS-DQN), which combines rich feature representations learned by
a DRL algorithm with the stability of a linear least squares method. We do this
by periodically re-training the last hidden layer of a DRL network with a batch
least squares update. Key to our approach is a Bayesian regularization term for
the least squares update, which prevents over-fitting to the more recent data.
We tested LS-DQN on five Atari games and demonstrate significant improvement
over vanilla DQN and Double-DQN. We also investigated the reasons for the
superior performance of our method. Interestingly, we found that the
performance improvement can be attributed to the large batch size used by the
LS method when optimizing the last layer.

本研究提出了一种名为最小二乘深度 Q 网络 (LS-DQN) 的混合方法，它将由深度强化学习算法学习的丰富特征表示与线性最小二乘方法的稳定性相结合。实验表明，与原始的深度 Q 网络 (DQN) 和 Double-DQN 相比，LS-DQN 在五个 Atari 游戏中均表现出显著的性能提升，此方法优越的性能主要归功于优化最后一层时所使用的大批量线性最小二乘方法。